Studies in Higher Education

ISSN: 0307-5079 (Print) 1470-174X (Online) Journal homepage: http://srhe.tandfonline.com/loi/cshe20

The development, validation and application of

the Course Experience Questionnaire

Keithia L. Wilson , Alf Lizzio & Paul Ramsden

To cite this article: Keithia L. Wilson , Alf Lizzio & Paul Ramsden (1997) The development,
validation and application of the Course Experience Questionnaire, Studies in Higher Education,
22:1, 33-53, DOI: 10.1080/03075079712331381121

To link to this article: https://doi.org/10.1080/03075079712331381121

Published online: 05 Aug 2006.

Submit your article to this journal

Article views: 2500

Citing articles: 194 View citing articles

Full Terms & Conditions of access and use can be found at

http://srhe.tandfonline.com/action/journalInformation?journalCode=cshe20
Studies in Higher Education Volume 22, No. 1, 1997 33

The Development, Validation and

Application of the Course
Experience Questionnaire
KEITHIA L. W I L S O N & ALF LIZZIO
School of Applied Psychology, Griffith University, Australia

PAUL R A M S D E N
Griffith Institute for Higher Education, Gnffith University, Australia

ABSTRACT The Course Experience Questionnaire (CEQ) is a development of work originally

carried out at Lancaster University in the 1980s. It is used as a measure of perceived teaching quality
in degree programmes in national annual surveys of all graduates in the Australian higher education
system and is increasingly being employed as a measure of the quality of teaching in universities in
the UK. This article discusses the development and use of the CEQ and the construction of a new
generic skills scale. Exploratory and confirmatory factor analyses of large multidisciplinary samples
of students and graduates from several universities established the reliability and validity of both the
full and short forms of the instrument, and identified a two-factor higher order structure. The criterion
validity of the instrument was also established, with scores on the CEQ demonstrating positive
correlations with students' approaches to learning, perceived course satisfaction, academic achieve-
ment and reported generic ('enterprise') skills development. The instrument also demonstrated
discriminant validity via its capacity to differentiate between pedagogically distinct programmes.
These results confirm the validity and usefulness of the CEQ as a performance indicator of university
teaching quality.

Introduction

The widespread demands from both governments and consumers for greater accountability
in higher education have been widely documented. These, in conjunction with the quality
improvement movement, have resulted in a need for valid, reliable and comparable perform-
ance data on teaching quality. The Course Experience Questionnaire (CEQ) (Ramsden,
1991) was designed as a performance indicator (PI) of teaching effectiveness, at the level of
whole course or degree, in higher education institutions and is a development of work
originally carried out at Lancaster University in the 1980s. The CEQ is based on a theory of
university teaching and learning in which students' perceptions of curriculum, instruction
and assessment are regarded as key determinants of their approaches to learning and the
quality of their learning outcomes (Marton & S/~lj6, 1976; Entwisfle & Ramsden, 1983;
Ramsden, 1992). The instrument was designed to measure differences in the quality of
teaching between comparable academic organisational units in those important aspects of
teaching about which students have direct experience and are therefore validly able to
comment (viz. quality of teaching, clear goals and standards, workload, assessment, emphasis

0307-5079/97/010033-21 © 1997 Societyfor Research into Higher Education

34 K . L . Wilson et al.

on independence). Student evaluations have been established as valid, reliable and useful
indicators of teaching quality (see Marsh [1987] for a review) and have the added value and
appealing benefit of being a direct measure of consumer satisfaction with higher education
(Ramsden, 1991).

Development of the Course Experience Questionnaire

The development of the CEQ and its theoretical and empirical basis have been previously
outlined in some detail (Ramsden, 1991). Development work used an item pool derived from
the Course Perceptions Questionnaire (Ramsden & Entwistle, 1981), a subsequent School
Experience Questionnaire (Ramsden et al., 1989), Experiences of Studying and Higher
Education Questionnaire (Entwistle & Tait, 1990) and items developed from an analysis of
open-ended student feedback. Based on the strength of preliminary studies, the Australian
Higher Education Performance Indicators Research Project (see Linke, 1991) recommended
a national trial of a 30-item version of the CEQ (CEQ30). This instrument comprised the
five scales of Good Teaching (8 items), Clear Goals and Standards (5 items), Appropriate
Workload (5 items), Appropriate Assessment (6 items) and Emphasis on Independence (6
items).
Ramsden (1991) reports the results of this national trial in which, using a large
cross-institutional sample, the CEQ30 scale structure was broadly confirmed. Scale validity
was determined by establishing relationships between scale totals and the external criteria of
quality of student learning, overall course satisfaction ratings and consistency with lecturers'
self-ratings of teaching effectiveness. Correlations between the five C E Q scales and the
Approaches to Studying Inventory (Entwistle et al., 1979) indicated a relationship between
heavy workload and inappropriate assessment and superficial reproductive study methods,
and between good teaching and clear goals and standards, and deep approaches aimed at
understanding.
Only two subsequent studies have reported findings on the structure of the CEQ30.
Trigwell & Prosser (1991), using a sample of 55 final year nursing students, investigated the
relationship between students' perceptions of their learning environment (using the CEQ30),
approaches to study and learning outcomes. High quality learning outcomes were related to
a deep approach to learning and perceptions of good teaching, clear goals and standards and
an emphasis on independence in learning. While scale factor loadings were less than those
reported by Ramsden (1991), these results broadly confirmed the scale structure of the
CEQ30. Richardson (1994) investigated the factor structure of the CEQ30 using a sample of
89 students from one academic department in a British institution. While this study also
broadly confirmed the scale structure of the CEQ30, the solution was somewhat problematic.
Eight out of 30 items were found not to contribute to the factor structure at all, and five of
the six items defining appropriate assessment loaded primarily on different scales.
The confidence with which both of these studies can be regarded is limited by the small
size of their samples. While the popular rule that sample size be determined as a fimction of
the number of variables (usually on a 10 subjects to 1 variable ratio) has been challenged
(Barrett & Kline, 1981; Velicer et aL, 1982), a minimum sample size of 150 has been
empirically demonstrated as necessary for moderate factor loadings (0.60), and a range of
300-400 for weaker loadings (0.40) in order confidently to generalise the results to a wider
population (Guadagnoli & Velicer, 1988).
It was proposed in the present study to address the problem of generalisability by
cross-validating the structure of the CEQ across several large multidisciplinary samples.
Apart from the limitations of sample size, there are obvious difficulties in comparing the
 Validation of the Course Experience Questionnaire 35

results of studies with different sample characteristics, and in drawing conclusions regarding
the structure of the CEQ. One aim of the present study was to overcome this sampling
problem by also applying a series of analyses to each individual sample. Thus, a convergent
methodology was employed with both exploratory and confirmatory factor analytic proce-
dures applied to analysing the structure of each sample.

Criticism of the CEQ

People will only have confidence in CEQ data to the extent that they have confidence in the
instrument itself. With the aim of developing a climate of open discussion and shared
ownership, the Graduate Careers Council of Australia (GCCA) has convened regular
national symposia in order to provide universities with a means of discussing both the
methodology and results of national CEQ surveys. While the CEQ as a performance
indicator has, for the most part, been well received, some criticisms have been raised
concerning the survey methodology, the instrument and the raters themselves.
The reliability of graduates as judges of course quality is still occasionally questioned but
with decreasing frequency. There is an increasing acceptance, based on empirical evidence
(Marsh, 1987), that student evaluations are valid and reliable indicators of teaching quality.
The variable response rates achieved by the CEQ national graduate survey, both within and
between institutions, is an ongoing challenge to the validity of comparisons. While response
rates for the 1992 national survey were reasonable (an average of 54%, ranging from 33% to
89% between universities (Ainley & Long, 1994), there is obvious room for improvement.
It has been suggested that the CEQ is 'psychometrically correct' because of its use of a
scale structure, but that it achieves this at the cost of being able to provide only a limited
coverage of the student experience (Yorke, 1995). The perceived quality of facilities (com-
puting, library) and support services (enrolment information) in particular are identified as
omissions. Yorke reports the development of a student experience questionnaire which uses
a 'broad brush' approach, where breadth of coverage is achieved by using single items, as
opposed to scales, to measure aspects of student experience. While a case can be made for
a broader sampling of aspects of student experience, it is interesting to note, however, that
in a survey of students across all years at one Australian university, satisfaction with facilities
was a much weaker predictor of overall student satisfaction than the current scales of the
CEQ (Ramsden, 1995).
Given the dynamic nature of higher education and our current steep learning curves
about how best to measure 'teaching quality', there is little possibility that the CEQ will be
immutable over time--modifications are a necessary outcome of a participatory process
which seeks to address the perceptions and needs, and thus earn the confidence, of
institutions. However, while accepting that the instrument will evolve over time in response
to need, we would advise against fundamental changes without careful consideration of the
implications for comparability of data over time. The existence of time series data is proving
to be one of the most useful aspects of the national CEQ data base.

Current Form of the CEQ

The most widely used version of the CEQ is the 23-item short form (CEQ23). This form was
developed in consultation with the Department of Employment, Education and Training
(DEET) and has been used in the G C C A national survey of graduates from 1993 onwards.
The strongest loading items from Ramsden's (1991) analysis of the original CEQ30 item
scale were retained to define the scales of Good Teaching (6 items), Clear Goals and
36 K.L. Wilson et al.

Standards (4 items), Appropriate Workload (4 items) and Appropriate Assessment (3 items).

The Emphasis on Independence scale (given its comparatively weaker scale structure) was
not included in this short form, and a new scale measuring Generic Skills (6 items) was
added.

Development of the Generic Skills Scale

In recent times there has been an increasing awareness and acceptance within higher
education of the need to produce graduates who are not only competent in the content of
their field of study, but also who possess process skills relevant to employability and lifelong
learning. In the UK, the Enterprise in Higher Education (EHE) initiative aims at developing
enterprise skills and attitudes in students--process skills which help students effectively to
apply the content or subject skills learnt in higher education to work environments (Macnair,
1990). Similarly, the Higher Education for Capability programme (Weil, 1992) seeks to
implement broader notions of 'capability' to include not only the acquisition of knowledge,
but also the use of knowledge to innovative and productive ends.
In Australia, several key reports have reflected similar concerns and aspirations. The
Higher Education Council paper Higher Education: the challenges ahead (HEC, 1990) empha-
sised the need for graduates to acquire a range of transferable, adaptive generic skills in a
society with dynamic patterns of work and changing knowledge bases. Perhaps most impor-
tantly, the Higher Education Council paper Achieving Quality (HEC, 1992) emphasised the
social relevance of education, and articulated the purposes of higher education as the
preparation of graduates for community leadership and the creation and application of
knowledge for the betterment of communities. The report summarised the results of several
studies which identified the skills which graduates and employers considered generic to
workplace competence.
The domain of such generic competencies is potentially quite extensive. The Generic
Skills scale of the CEQ is designed to provide a brief, uniform national measure by which
institutions can assess the extent to which graduates perceive their courses as developing a
number of generic skills and abilities--problem-solving, analytic skills, teamwork, confidence
in tackling unfamiliar situations, ability to plan work and written communication skills. The
Generic Skills scale was included in the 1992 Graduate Destination Survey, and data on
graduate perceptions of generic skill development in Australian institutions by field of study,
level of course and gender have been collated (Ainley & Long, 1994). However, no findings
to date on the validity or reliability of the Generic Skills scale have been reported. This is one
of the aims of the present study.

Purpose of the Present Study

Further investigation of the CEQ is suggested as a priority given its widespread use in the
Australian higher education system, its early use in some institutions in the UK, and the
relatively few studies reporting on its reliability and validity. The present study sought to
achieve the following five outcomes:

1. to provide further data on the construct validity of both the tong ([CEQ36]
CEQ30 + Generic Skills scale) and short (CEQ23) forms of the CEQ, using exploratory,
factor analysis with a number of large cross-disciplinary samples of students and gradu-
ates;
 Validation of the Course Experience Questionnaire 37

2. to provide, using confirmatory factor analysis, a more stringent assessment of the validity
of both the long and short forms of the CEQ;
3. to establish the reliability and validity of the new Generic Skills scale;
4. to provide further data on the criterion validity of the CEQ as a measure of teaching
quality by establishing the relationship between scale scores and levels of approaches to
learning, overall course satisfaction and academic achievement; and
5. to provide discriminant validity data on the capacity of the CEQ to discriminate between
pedagogically distinct programmes in a field of study (e.g. to discriminate between
problem-based and traditionally taught courses).

Method

Samples and Procedure

Three broad-based surveys of students from the same university were conducted. Participants
in the 1993 survey of students were randomly selected to represent approximately equal
numbers of males and females across 14 faculties (including humanities, business, commerce,
environmental sciences, computing sciences, social sciences, law, education, health sciences,
and visual and performing arts) and all years of study. For the 1994 survey, all undergraduate
students (Years 1-3) were sampled. In order to test the stability of the CEQ structure across
both student and graduate populations, all 1992 graduates of the university were also
sampled. For all three samples, questionnaires were mailed to individual students' home
addresses, approximately 3 months after completion of the academic year, to be returned
directly to the research project base within the university. Up to three follow-up letters were
sent in order to achieve a minimum response rate of 30% within each faculty, and 50%
overall for the university. For the 1992 graduate sample, a total of 1362 usable responses
(response rate 54%) was obtained. Biographical information was not available for this
sample. For the 1993 student sample, a total of 2130 usable responses (response rate 54%)
was obtained. The sample comprised 1043 female (49%) and 1087 male (51%) students. For
the 1994 student sample, a total of 7370 usable responses (response rate 55%) was obtained.
The sample comprised 4415 (60%) female and 2955 (40%) male students.
Given the widespread use of the CEQ, samples of graduates from other Australian
universities participating in the 1993 and 1994 G C C A surveys were also used in this study.
Characteristics of these samples are provided later in this article.

Materials
Questionnaires included two versions of the CEQ (Ramsden, 1991). The full form (36 items;
CEQ36) containing all six scales was administered to the 1992 graduate, and 1994 student
samples. The 23-item shorter form (CEQ23) containing the full version of the Generic Skills
scale, shortened versions of the Good Teaching, Appropriate Workload, Appropriate Assess-
ment and Clear Goals and Standards scales, and excluding the Emphasis on Independence
scale was administered to the 1993 student sample (see appendix for the items relevant to all
forms of the CEQ).
In order to establish the relationship between students' evaluations of teaching as
measured by the CEQ and approaches to learning, two short scales (6 items each) were
constructed, using items from the Approaches to Studying Inventory (Entwistle et al., 1979)
to represent deep and surface approaches. Both the reliability and validity of these short
scales were established using data from the 1992 sample. Both scales evidenced moderate
38 K. L. Wilson et al.

TABLE I. Coefficient alpha values from Ramsden's (1991), Richardson's (1994) and the
present

Present study

1993 1994 1992

Ramsden Richardson student student graduate
CEQ scale (1991) (1994) sample sample sample

Good Teaching 0.87 0.79 0.86 0.88 0.88

Clear Goals 0.80 0.77 0.82 0.82 0.76
Workload 0.77 0.71 0.75 0.74 0.69
Assessment 0.71 0.47 0.74 0.73 0.70
Independence 0.72 0.55 0.68 0.67 --
Generic Skills -- -- 0.80 0.79 0.77

levels of i n t e m a l consistency ( C r o n b a c h ' s alpha coefficients of 0.67 for deep approach and
0.69 for surface approach), which are in excess of those typically reported for these subscales
(Entwistle et aL, 1979; Entwistle & Kozeki, 1985), and close to the limit of 0.71 suggested
by C o m r e y (1973) as indicating acceptable i n t e m a l consistency. Construct validity was
confirmed b y means of a principal c o m p o n e n t s factor analysis with a varimax rotation on the
12-item set. T w o distincx factors accounting for 4 0 % of the variance (factor 1 = 24%; factor
2 = 16%) were clearly identifiable, with all six deep items and all six surface items loading in
excess o f 0.30 on the first a n d second factors respectively. A small b u t significant negative
correlation ( - 0 . 2 1 , p < 0 . 0 0 1 ) between the two scales further supports their construct
validity as measures o f deep and surface approaches to learning.
A c a d e m i c achievement was m e a s u r e d by averaging each student's grade p o i n t average
(GPA), m e a s u r e d on a scale from 1 to 7, from the c o m m e n c e m e n t of their degree to the point
at which the survey was conducted. This data (available only for the 1992 sample) was
obtained from official university records.

Results and D i s c u s s i o n

Extensive statistical analyses were c o n d u c t e d to assess the psychometric characteristics of the

C E Q as a measure of the perceived quality o f teaching. All statistical analyses were c o n d u c t e d
using the p r o g r a m S P S S for W i n d o w s Version 6.0 (Norusis, 1993).

Reliability of the CEQ

T h e reliability o f each scale of the C E Q was m e a s u r e d using C r o n b a c h ' s (1951) coefficient
alpha. Alpha coefficients for all three samples indicate m o d e r a t e to high levels o f internal
consistency for all scales (see T a b l e I). Reliability coefficients for the original five scales were
consistent with those from R a m s d e n ' s (1991) pilot study and generally higher than those
from Richardson's (1994) study, especially for the Emphasis on I n d e p e n d e n c e scale. T h e
alpha coefficients for the short form (CEQ23; 1992 graduate sample), while marginally lower
on some scales c o m p a r e d to the full form (CEQ36; 1993 and 1994 student samples),
nevertheless d e m o n s t r a t e acceptable levels of internal consistency. Thus, b o t h forms can be
considered reliable instruments.
 Validation of the Course Experience Questionnaire 39

Construct Validity of the CEQ--exploratory factor analysis

Item and scale exploratory factor analyses (EFAs) were conducted on all samples using
principal components factor analysis with an oblique rotation. The choice of principal
components factor analysis provided a basis for direct comparison with Ramsden's (1991)
pilot study. The oblique method of rotation is recommended where items are assumed to be
correlated with each other (see Tabachnick & Fidell, 1989). Given the history of significant
positive and moderate intercorrelations demonstrated between the CEQ scales (Ramsden,
1991; Ainley & Long, 1994), an oblique rotation was assumed to be most appropriate. A
combination of the scree test (Cattell, 1966) and the eigenvalue greater-than-one rule
(Kaiser, 1974) was used for determining the appropriate number of factors to be extracted.
Only factor loadings of 0.30 and above were selected for interpretation.

Item Facwr Analysis

Long form of the questionnaire--CEQ36. Item factor analyses of both the 1993 and 1994
student samples confirmed firstly the original five factor structure reported by Ramsden
(1991), and secondly, the structure of the new Generic Skills scale. A stable and virtually
identical six factor solution accounting for 52% of the variance was produced in both
samples: factor 1, Good Teaching (27%; 28%); factor 2, Generic Skills (7%; 7%); factor 3,
Appropriate Workload (5%; 5%); factor 4, Appropriate Assessment (5%; 5%); factor 5,
Clear Goals and Standards (4%; 4%); and factor 6, Emphasis on Independence (3.5%; 3%).
Table II presents the pattern of significant item loadings for both samples.
The first factor comprised all eight items from the Good Teaching scale, plus two
additional items (7, 29) from the Appropriate Assessment scale and one item (30) from the
Emphasis on Independence scale. The second factor comprised all six items from the Generic
Skills scale, and one additional item (15) from the Emphasis on Independence scale. The
third factor comprised all five items from the Appropriate Workload scale, and an additional
item (32) from the Appropriate Assessment scale. The fourth factor comprised four of the six
items from the Appropriate Assessment scale. The fifth factor comprised the five items
defining the Clear Goals and Standards scale. The sixth factor comprised four of the six items
from the Emphasis on Independence scale. Both samples produced factor structures which
dearly identified all six scales of the CEQ.
In contrast to Richardson's (1994) finding that eight items of the C E Q failed to load on
any factor, all items in both samples of the present study loaded on one of the six factors.
Only two items loaded significantly on more than one factor. On both samples, item 32 ('it
would be possible to get through this course just by working hard around exam times') loaded
most strongly (0.63) on its designated scale of Appropriate Assessment and also, to a lesser
extent (0.31, 0.33), on the Appropriate Workload scale.
Three items (15, 7, 29) loaded on factors different to their nominated CEQ scale. Item
15 loaded on the Generic Skills factor, rather than the Emphasis on Independence factor,
suggesting that students perceived the 'development of one's own academic interests' to be
one of the generic competencies acquired through a university education. The two items on
the Appropriate Assessment scale (7, 29) related to the motivational and interpersonal
feedback process associated with assessment (staff leaming from students and providing
qualitative feedback in addition to marks) rather than the type of assessment in a course,
loaded less than 0.30 on the Assessment factor, and instead, loaded more strongly on the
Good Teaching factor. This suggests that such 'good teaching behavioLlr' can obviously be
evident in courses with otherwise inappropriate reproducing or surface approaches to assess-
 TABLE I I . R e s u l t s o f f a c t o r a n a l y s e s o n C o u r s e E x p e r i e n c e Q u e s t i o n n a i r e item scores for three samples

Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6

Item C E Q scale 1993 1994 1992 1993 1994 1992 1993 1994 1992 1993 1994 1992 1993 1994 1992 1993 1994 1992

25 Good Teaching 0.62 0.54 0.72

23 ,, 0.56 0.54 0.71
4 0.65 0.52 0.63
20 ,, 0.71 0.66 0.82
31 ,, 0.51 0.51 X X X X X X m
33 ,, 0.41 0.36 X X X X X X ""
9 ~ 0.67 0.68 0.76
22 ,, 0.69 0.71 0.80

12 Generic Skills 0.82 0.81 0.73

2 0.79 0.75 0.76
6 0.76 0.76 0.74
13 0.68 0.73 0.67
28 ,, 0,66 0.67 0.65
11 ,, 0.46 0.51 0.42 0.37

5 Worldoad 0.78 0.77 0,79

27 ,, 0.76 0.72 0.73
36 ,, 0.72 0.68 0.70
19 ,, 0.49 0.59 0.52
14 ,, X X 0.50 0.53 X X X

10 Assessment 0.73 0.74 0.76

17 0.69 0.65 0.73
32 ,, X X 0.33 0,31 X 0.63 0.63 X X X
26 0.58 0.58 0.66
29 ,, 0.56 0.56 X X X X X X
7 ~ 0.45 0.43 X X X X X X

1 Clear Goals 0.78 0.75 0.79

18 ,, 0.76 0.73 0.70
8 ~ 0.75 0.77 0.71
35 ,, 0.32 0.70 0.72 0.63
24 ,, X X X 0.64 0.64 X

16 Independence X X X X X 0.67 0.65 X

21 X X X X X 0.68 0.63 X
34 ,, X X X X X 0.55 0.59 X
3 X X X X X 0.51 0.58 X
30 ,, 0,44 0.51 X X X X X 0.30 X
15 ,, X 0.43 0.46 X X X X X

Factor loadings ~ 0.30 shown; X = item n o t i n c l u d e d on 1992 graduate survey.

 Validation of the Course Experience Questionnaire 41

ment. Similarly, the single item from the Independence scale which focused on student-
teacher interaction (item 30: 'We discuss with staff how we are going to learn') loaded more
strongly on the Good Teaching (0.44, 0.51) than its nominated Emphasis on Independence
factor (0.30). It should be noted that these cross-loading items are those eliminated from the
original version to create the 23-item short form of the CEQ.

Short form of the questionnaire--CEQ23. The results for the short form were very similar to
those for the long form of the CEQ. A five factor solution accounted for 57% of the variance:
factor 1, Good Teaching (27%); factor 2, Appropriate Workload (11%); factor 3, Generic
Skills (7%); factor 4, Appropriate Assessment (7%); and factor 5, Clear (Goals and Stan-
dards (5%) (see Table III). The pattern of factor loadings provided clear identification of all
five CEQ scales. All 23 of the items loaded on their nominated scales. Two items evidenced
cross-loading on to a second scale. While item 35 (clear expectations) loaded most strongly
on its designated scale of Clear Goals and Standards, it also loaded moderately on Good
Teaching. Similarly, while item 11 (development of skills as a team member) loaded most
strongly on to Generic Skills, it also loaded moderately on to Appropriate Assessment. This
may reflect the increasing trend towards team- or group-based assessment practices. This
23-item shorter form of the CEQ offers a stable factor structure equal to that of the 36-item
full form, with the advantage of cleaner relationships between items and scales.

Construct Validity of the CEQ--confirmatory factor analysis

The construct validity of the CEQ36 and CEQ23 was tested further through the application
of structural equation modelling techniques (EQS) (Bentler, 1989). Confirmatory factor
analysis (CFA) was used to test the item to scale structure, and higher order path analysis was
used to test the higher order scale structure. CFA permits the direct testing of the CEQ as
a theoretical model through the calculation of a measure of fit between the proposed model
of the designated CEQ scales, and the optimal model derived from estimates of the various
relationships. Thus, in contrast to EFA which provides an indirect test of a theoretical model,
CFA tests the model directly (Bernstein & Teng, 1989). Both the long and short forms of the
CEQ were tested with the same samples as those used for the EFAs.
EQS provides three goodness-of-fit indices as a guide to interpretation. These include
chi-square, the comparative fit index (CFI) (the agreement between the empirical results and
the results expected by the model), and the residual mean square (RIMS) (an index of the
degree to which the initial correlation matrix is not reproduced by the estimated factor
model). A good fit is generally indicated by a low and non-significant chi-square value, a CFI
in excess of 0.8 (Cole, 1987) or 0.9 (Bentler & Bonett, 1980) (the maximum value is 1.0),
and an RMS not higher than 0.05 (Kerlinger & Pedhazur, 1973). However, the interpretation
of the chi-square index should be made in the context of sample size, since large sample sizes
inflate chi-square values. Thus, with large sample sizes, significant chi-squares do not
necessarily indicate a poor fit of the model to the data (see Saris & Stronkhorst, 1984;
Wheaton, 1987; Pedhazur & Pedhazur Schmelkin, 1991). In fact, Wheaton (1987) argues
against its use as an index of fit for this reason.

Item factor analysis. The results of the item confirmatory analyses revealed a moderate
overall fit of the data to the model for the long form of the CEQ (chi-square = 3865.72,
p < 0.001; CFI = 0.85; normed fit index = 0.84; non-normed fit index = 0.83; based on 579
degrees of freedom; RMS = 0.04) and a good fit for the short form (chi-square = 1221.16,
p < 0.001; CFI = 0.90; normed fit index = 0.89; non-normed fit index = 0.89; based on 220
42 K . L . Wilson et al.

degrees of freedom; RMS = 0.04). The inflated chi-square values are clearly attributable to
the large sample sizes. Not surprisingly, the short form model (CEQ23) offers a better fit than
the long form model (CEQ36). The difference in fit between the two forms can be accounted
for largely by the Emphasis on Independence factor on the long form where a number of
items evidencing low factor loadings (range of 0.25 to 0.60) and high structural coefficients
for the error term (range of 0.80 to 0.97) detract from the overall goodness of fit of the
model. This demonstrated improvement in fit is consistent with Bentler & Bonett's (1980)
proposition that models with overall fit indices of less than 0.9 can usually be improved
substantially.
All factors or scales on both the long and short forms were confirmed. For the short
form, the factor loadings for each item ranged from 0.70 to 0.79 for the Good Teaching scale,
0.61 to 0.70 for the Clear Goals and Standards scale, 0.56 to 0.85 for the Appropriate
Assessment scale, 0.48 to 0.71 for the Appropriate Workload scale and 0.35 to 0.71 for the
Generic Skills scale. The structural coefficients for the error terms were generally modest.
The only exceptions were one item each from the Generic Skills (item 11), Appropriate
Workload (item 19) and Appropriate Assessment (item 10) scales, which combined lower
factor loadings and higher structural coefficients, indicating the potential for further improve-
ment of these scales.
Given the widespread use of the CEQ23 (i.e. the short form), it was considered
important to confirm further its structure using samples of graduates from other universities.
Confirmatory factor analyses of two large multi-disciplinary samples, one from a traditional
university (n = 3561), and one from a newer university with a more applied focus (n = 3080),
confirmed the item to scale CEQ23 model, viz. traditional university--chi-square = 4175.79,
p < 0.001; CFI = 0.87 (normed fit index = 0.87; non-normed fit index = 0.85) based on 220
degrees of freedom; RMS = 0.05; and applied university--chi-square = 2576.99, p < 0.001;
CFI = 0.89 (normed fit index = 0.89; non-normed fit index = 0.88) based on 220 degrees of
freedom; RMS = 0.04.

Scale factor analysis. Can the item scores on the CEQ be aggregated to yield a single global
score of teaching quality? Trigwell & Prosser's (1991) factor analysis of scale scores produced
a single higher order factor. Richardson's (1994) analysis of the CEQ30 structure produced
a solution which he describes as 'a monarchical hierarchy' (Cattell, 1965): correlated but
non-overlapping first order factors dominated by a single second order factor. Richardson
(1994) concludes that a single global measure of teaching quality can be derived from the
CEQ. However, Ramsden (cited in Richardson, 1994), suggests that the CEQ may comprise
two higher order factors, one reflecting good teaching and assessment, and the other
reflecting an appropriate workload. This proposition is consistent with the Appropriate
Workload scale consistently demonstrating lower factor loadings on a single higher order
factor relative to the other scales (Trigwell & Prosser, 1991; Richardson, 1994). Three higher
order models were tested by means of higher order path analysis--a one-factor (all scales), a
two-factor (Appropriate Workload and all other scales), and a three-factor (Appropriate
Workload, Generic Skills and all other scales) higher order scale structure.
Results indicated a moderate fit for the one-factor, and a good fit for the two- and
three-factor solutions on both the long (CEQ36) and short (CEQ23) forms (see Table III).
Whereas the separation of Appropriate Workload and the other scales (in the two factor
model) offers an improvement in fit over a one-factor solution, the separation of Generic
Skills (in the three-factor model) does not offer an improvement of the two-factor model.
Thus, it is suggested that the higher order structure of the CEQ can be most usefully
 Validation of the Course Experience Questionnaire 43

TABLE III. Goodness-of-fitindicators for higher order path analysis on two samples

Long form CEQ36: Short form CEQ23:

1993 student sample 1992 graduate sample

Index 1 factor 2 factor 3 factor 1 factor 2 factor 3 factor

Chi-square 4029.04 57.23 48.43 13t9.40 47.26 39.14
p<0.001 p>0.05 p>0.05 p<0.001 p>0.001 :o>0.05
Comparative fit index 0.84 0.98 0.98 0.89 0.95 0.96
Normed fit index 0.83 0.98 0.98 0.88 0.95 0.96
Non-normed fit index 0.82 0.96 0.96 0.88 0.87 0.78
Degrees of freedom 581 8 6 219 4 2
Residual mean square 0.05 0.02 0.01 0.05 0.02 0.02

conceptualised as comprising two-factors--good teaching, clear goals and standards, generic

skills and appropriate assessment; and appropriate workload.

Criterion Validity of the CEQ--relationship between CEQ and external criteria

A series of correlational analyses were conducted to test the degree of relation between scores
on the scales of the CEQ and a number of key external criteria related to teaching and
learning effectiveness. Biggs's (1979, 1985) three stage model of student learning involving
the interaction of presage (aspects of the learning environment measured by the CEQ),
process (approaches to learning) and product (learning outcomes such as satisfaction,
academic achievement and generic skills) variables was used to conceptualise this analysis of
construct validity.

Presage to process criteria--relationship between CEQ ratings and deep and surface approaches to
learning. Correlational analyses, using data from the 1992 sample, were conducted between
students' perceptions of the learning environment (measured by the scales of the CEQ) and
reported approaches to learning (measured by deep and surface subscales of the Approaches
to Studying Inventory). All the CEQ scales evidenced significant positive correlations with a
deep approach, and significant negative correlations with a surface approach to learning (see
Table IV). A deep approach to student learning (emphasis on understanding and deriving
meaning) was related most strongly to good teaching, appropriate assessment and indepen-
dence in learning. In contrast, a surface approach to student learning (emphasis on reproduc-
ing facts) was most closely related to heavy workload and inappropriate assessment. While a
number of the correlations were small, but statistically significant owing to the large sample
sizes, these results are nevertheless consistent with a pattern of previous findings at both an
individual and course level in secondary and tertiary contexts using the CEQ, its forerunner
the Course Perceptions Questionnaire, and the School Experiences Questionnaire (Entwistle
& Ramsden, 1983; Kember & Gow, 1989; Meyer & Parsons, 1989; Ramsden et al., 1989;
Entwistle & Tait, 1990; Ramsden, 1991; Trigwell & Prosser, 1991; Eley, 1992; Gibbs &
Lucas, 1995). Clearly, then, the CEQ is measuring aspects of the teaching environment
which are systematically associated with students' reported learning processes.

Presage to product criteria--relationship between CEQ ratings and course outcomes. Based on the
proposition that more effective courses will produce greater student satisfaction, higher
academic achievement and higher generic skills development, correlational analyses were
conducted to investigfite the relationship between the CEQ scale scores and the external
TABLE IV. Correlations b e t w e e n deep a nd surface approaches to studying a nd C E Q scales for Trigwell & Prosser (cited in
R a m s d e n , 1991) a n d 1993 s a m p l e in the pre s e nt s t udy

D e e p a pproa c h Surface a pproa c h

Trigwell & Prosser Present study Tri gw e l l & Prosser Present study
C E Q scale (1991) 1993: s t u d e n t sample (1991) 1993: s t u d e n t samp le

G o o d teaching 0.15 0.24"** - 0.10 - 0.34***

Clear goals 0.10 0.12"** - 0.24 - 0.29***
W o rk lo ad 0.04 0,07** - 0.45 - 0.48***
Assessment 0.17 0.21"** - 0.43 - 0.47***
Independence 0.02 0.19*** - 0.27 - 0.29***
Generic Skills -- 0.37*** --- - 0.20***

n = 2130; **p < 0.01; ***p < 0.001.

 Validation of the Course Experience Questionnaire 45

criteria of (a) students' overall satisfaction with their courses for all three samples; (b) generic
skills development for all three samples; and (c) academic achievement for the 1993 student
sample. Significant positive correlations were found on all samples between all scales of the
CEQ and overall course satisfaction, academic achievement, and generic skills (see Table V).
The correlations with course satisfaction were consistent with, and stronger than, those in
Ramsden's (1991) original study. The Good Teaching and Clear Goals and Standards scales
correlated most strongly with satisfaction and academic achievement, and the Good Teach-
ing and Emphasis on Independence in Learning scales with Generic Skills. The Appropriate
Workload scale demonstrated the lowest correlations with satisfaction, academic achievement
and generic skills. The positive association between scores of the CEQ and the measures of
learning outcome--satisfaction, academic achievement and generic skills--further strength-
ens the instrument's validity as a measure of teaching quality.

Discriminant Valid#y of the CEQ--can the CEQ discriminate between courses with different
teaching philosophies and methods?
Richardson (1994) proposed that a further test of the CEQ would be an assessment of its
capacity to discriminate between courses in terms of their explicit objectives. In order to test
its discriminant validity, CEQ profiles of universities participating in the 1993 and 1994
national surveys of graduates were compared in two fields of study (medicine and psy-
chology) where there were clear examples of programmes with distinct course objectives and
teaching philosophies. One of the 10 medical programmes in the 1993 and 1994 surveys was
known to be conducted along problem-based lines, and one of the 34 psychology pro-
grammes in the 1993 survey, and 31 psychology programmes in the 1994 surveys was known
to be conducted along experiential and action learning lines. Only those departments with a
sample size in excess of 15 (which assumes a normal distribution) were included for analysis.
Analyses of variance were conducted on CEQ scale scores for each of the 10 medical
programmes for 1993 and 1994. While there was no significant difference between the
programmes on the Good Teaching scale, the pattern of differences on other scales was
consistent with those expected when comparing problem-based and traditionally taught
courses. CEQ scores on the Clear Goals and Standards scale were significantly lower for the
problem-based courses than all other programmes in boil: the 1993 and 1994 surveys.
Generic Skills scale scores for the problem-based programme were the highest in 1993 (and
significantly higher than eight other programmes) and significantly higher than all other
programmes in 1994. The problem-based programme achieved second highest and highest
scores on the Appropriate Assessment scale in the 1993 and 1994 surveys respectively, which
were significantly higher than four other programmes in 1993 and eight in 1994.
Comparisons of the CEQ scale scores across 34 psychology programmes in 1993 and 31
in 1994 again demonstrated the instrument's capacity to discriminate pedagogically distinct
courses. The experiential psychology programme achieved the significantly highest scores on
Good Teaching in 1993, and the second highest in 1994; the significantly highest scores on
Appropriate Assessment in 1993 and the third highest in 1994; and the highest scores on
Generic Skills in both 1993 and 1994.

Conclusions about the CEQ as an Instrument

Present findings reinforce the confidence with which the CEQ can be used as an educational
evaluation tool. The structure of the long form of the instrument (CEQ36), incorporating
both the original five scales and the additional Generic Skills scale, demonstrates a high
degree of stability, and the short form (CEQ23) provides an equally stable structure.
 (5

v-

TABLE V. Correlations between the C E Q scales and course satisfaction, genetic skills and academic achievement (GPA) for the present study

Course satisfaction Genetic skills Academic achievement

1993 student 1994 student 1992 graduate 1993 student 1994 student 1992 graduate 1993 student
C E Q scale sample sample sample sample sample sample sample

Good Teaching 0.64 0.64 0.63 0.46 0.48 0.40 0.47

Clear Goals 0.55 0.55 0.50 0.33 0.30 0.29 0.46
Workload 0.33 0.36 0.17 0.16 0.15 0.02 0.24
Assessment 0.47 0.41 0.33 0.35 0.37 0.20 0.36
Independence 0.54 0.51 -- 0.41 0.40 -- 0.33
Genetic Skills 0.50 0.51 0.47 na na na 0.23

All values 0.15 and above are significant at p < 0.001.

 Validation of the Course Experience Questionnaire 47

Additionally, the CEQ would appear to measure constructs directly relevant to students'
reported approaches to, satisfaction with, and outcomes of, their learning in university
contexts. The CEQ's sensitivity to differences, along theoretically predictable lines, between
traditional and problem-based and experiential programmes would suggest its useful appli-
cation in research studies seeking to establish the comparative educational efficacy of learning
environments. The CEQ can thus be regarded as a valid, reliable and stable instrument.

Practical Applications
Use of the CEQ
Given its demonstrated validity as a performance indicator of' perceived teaching quality,
how is the CEQ being used currently in the Australian higher education system? The CEQ
is widely used as part of a national strategy of providing universities with system-wide
information which they can use to make informed judgements about the quality of the
courses they are offering (Ainley & Long, 1994). Since 1992, it has been distributed to all
Australian university graduates within a few months of course completion, as part of the
GCCA Graduate Destination Survey. The use of the CEQ as a standard national instrument
to measure graduates' perceptions of teaching quality has significant advantages for address-
ing the challenge of improving teaching quality. A nationally administered CEQ provides
comparative data to enable factually-based dialogue between institutions regarding what
constitutes 'best practice' in teaching. Individual institutions are able to work with a
data-based estimate of their relative performance, in comparison to similar institutions, both
as a whole institution and within a field of study (e.g. law, medicine). Additionally, national
field general trends (e.g. graduates' perceptions of law education across all Australian
universities) can be identified for field of study investigations and reviews by professional
bodies and government committees. Perhaps most importantly, the use of the CEQ as a
standard national instrument over a number of years will allow the accumulation of time
series data and the monitoring of change over time at various levels of academic organis-
ation--individual degree programme, institution, field of study and, indeed, the whole
national system.
Given the increasingly competitive higher education market-place, an understandable
temptation exists for university administrators to engage in simplistic cross-institutional
rankings of relative merit ('our CEQ scores are better than theirs'). However, greater
potential for long-term improvement and system-wide learning would appear to lie in a
co-operative strategy of inter-institutional benchmarking for best practice in fields of study.
The Australian Vice Chancellors' Committee (AVCC) is currently inviting universities to
nominate courses which they consider to be exemplars of excellent teaching, as the basis of
a national symposium for the dissemination of best practice.
A standard system for assessing teaching effectiveness can not only aid managerial, but
also consumer judgements of quality. CEQ national survey data are now also available in a
form that can be readily used by the consumers of higher education (i.e. potential students)
in making choices about what and where to study. The Good Universities Guide w Australian
Universities (Ashenden & Milligan, 1995), an annual evaluative and independent guide to the
quality of Australian universities, their campuses and courses, now includes course ratings by
recent graduates. Fields of study are compared across some CEQ scales using a five star
system (the more stars the better) based on scale averages. Degree programmes at individual
institutions are described as 'better', 'average' or 'worse' on teaching quality, workload and
overall satisfaction depending on whether recent graduates' ratings of that course were
48 K . L . Wilson et al.

significantly more or less favourable than graduates who had completed similar courses at
other institutions. The Good Universities Guide is careful not to endorse the credulous use of
such ratings. After presenting the possible limitations of such global comparisons, it advises
potential students that: 'the information in these ratings is significant and valuable, but must
be treated as one, and only one, among many pieces of information about courses and
campuses, and certainly must not be regarded as precise or infallible' (p. 6).
Publishing independent course ratings, with institutional approval, in the higher edu-
cation market-place would seem to indicate an increased willingness for Australian universi-
ties to regard their students as clients and consumers, and to be responsive and accountable
to their perceptions of course quality. Such data will become increasingly important if the
'user pays' principle is used to fund the expansion of postgraduate education in Australia.

Conditions for Effective Use of the CEQ

The potential for CEQ ratings to be used and misused in a variety of public forums (e.g.
university marketing, media stories) has led the AVCC through its Standing Committee on
Statistics, to draft a code of practice for the use and public disclosure of CEQ data (AVCC,
1995). If the full benefits of such a national strategy are to be realised, higher education
institutions will need to exercise caution in several areas: the use of the instrument, the types
of comparisons made, meanings attached to scores, and decision-making processes based on
CEQ data.
The CEQ is designed as a summative programme evaluation tool at the whole course/
degree level within an institution. It is not designed or intended to provide specific or
fine-grained diagnostic feedback regarding individual subjects or teachers. Such specific data
are best obtained through customised surveys and qualitative processes. While it is under-
standable that individual institutions would wish to use standard evaluation tools, the
frequent use of the CEQ for subject level evaluation will, apart from adding to already high
levels of 'questionnaire fatigue' among students, potentially compromise the results of
planned surveys.
While the national CEQ survey provides the data to make any number of comparisons
between and within institutions, only certain types of comparisons are legitimate. For
example, while comparisons across institutions within a particular field of study (e.g.
psychology) may be a useful and appropriate exercise, comparisons across fields of study
within an individual institution are problematic, given variation of course content and student
characteristics. Similarly, comparisons between universities, at the level of whole institution,
without awareness and acknowledgement of differences in historical and current circum-
stances, seems less than fair and useful. As Ainley & Long (1994) conclude, results provided
by the CEQ should be regarded as indicative rather than conclusive, and all comparisons
need to include an element of informed judgement.
Finally, scores on summative evaluation instruments such as the CEQ, while reliably
indicating areas of general teaching strength and weakness, do not in themselves provide a
sufficient basis for decision-making or intervention regarding specific improvements in
teaching quality. Institutions should regard a CEQ profile for a degree programme more as
a point of departure than an end point in an evaluation process, and use the survey results
to indicate productive directions for investigation into underlying factors affecting teaching
quality. Table VI summarises potential uses and misuses of the CEQ.
 Validation of the Course Experience Questionnaire 49

TABI~ VI. Uses and misuses of the CEQ in the measurement of teaching quality

Inappropriate use of the CEQ Appropriate use of the CEQ

Freqnent/regular use Intermittent planned use

Individual teacher/subject evaluation
Formative evaluation
Comparisong of programmes across fields of
study
Comparisons between dissimilar institutions
Inter-institutional ranking
As a single performance indicator of teaching
quality
Basis for decision-making regarding specific
changes in teaching
A unidimensional global measure of teaching
quality
Regarded as a direct measure of teaching
quality
Used to infer learning outcomes
Single criterion for student decision-making
about course enrolment
Results used only as a measure of 'customer
satisfaction
Results used for hard managerialist tactics
Whole course/degree programme evaluation
Summative evaluation
Cautions contextualised comparisons of
programme within a field of study
Cautious contextualised comparisons
between similar institutions
Benchmarking between institutions to
indentify possible best practice
As one of a range of performance indicators
of teaching quality
As an indicator of useful directions for further
investigation
A multidimensional assessment of teaching
quality
Regarded as a measure of perceived teaching
quality
Regarded as a measure of teaching process
Part of the basis for informed choice
Results used to improve student learning
outcomes
Results used to help staff make professional
judgements about how to improve
student learning outcomes

Examples of Good Practice

Several examples exist of 'good practice' where C E Q data have been used to stimulate
collaborative investigation into teaching and learning processes in institutions. G i b b s & Lucas
(1995) report such a use of the C E Q at Oxford Brookes University in what is termed a
'research orientation to development'. Gibbs and his colleagues collect data on class size,
patterns of assessment, student performance and C E Q ratings, and then identify areas where
'interesting patterns' emerge. Staff in these areas are then engaged as collaborative re-
searchers in trying to 'make sense' of the data. M c I n n i s (1995) reports the use of a similar
collaborative approach. M e l b o u r n e University is utilising the C E Q as part of a benchmarking
and improvement process investigating quality of teaching and learning from the student's
perspective in the first year of study. Staff are involved in developmental projects aimed at
enhancing student satisfaction and learning outcomes. Such approaches to programme
evaluation and development offer the joint outcomes of course improvement and academic
staff development. Clearly, while higher education institutions are working in contexts which
increasingly emphasise external accountability, approaches which attempt to stimulate and
utilise intrinsic interest in course development offer the most promise for self-sustaining
improvement.
O n e of the key challenges has been for institutions to find effective ways of presenting
C E Q survey data to academic staff. A relatively c o m m o n strategy is for a central administra-
tion unit (typically termed an Academic Services U n i t or P l a n n i n g and Development Unit)
to prepare a report for academic units (e.g. school, department or faculty). Written reports,
however, have had limited success as a method of creating change or improvement in
50 K.L. Wilson et al.

organisations. Part of the difficulty is that organisational reports often just 'present infor-
mation' rather than create a context which facilitates action.
Bate et al. (1995) have developed a promising reporting format for the dissemination of
CEQ survey data within an institution. Firstly, a working context is established for the user
in relation to the process of the CEQ survey--addressing questions regarding the validity of
the instrument, the purpose of the national strategy, guidelines for responsible interpretation,
and strategies for benchmarking best practice. Once the process of the survey has been
legitimated, the content of the CEQ results for a particular academic unit is presented in
context. Results are presented firstly for the institution as a whole, then profiles for individual
degree programmes, and finally comparative data are presented for degree programmes
offered by other institutions in the same field of study. Importantly, efforts are made to
differentiate which CEQ results may be national field of study effects (patterns common to
similar institutions in the same field of study), and which are specific institutional effects. The
CEQ data for an individual degree programme are then contextualised in relation to other
qualitative indicators (graduates' written comments on the programme) and quantitative
indicators (rates of completion, retention and progression, and employer satisfaction).
Finally, recommendations are made for further investigation and action.

Concluding Comments
This paper has sought to confirm further the validity and usefulness of the CEQ and to
outline conditions for its effective application as a performance indicator of university
teaching quality. In terms of its wider use, the CEQ's genesis in the U K higher education
system, its refined theoretical base, and its demonstrated measurement qualities indicate that
it could be generally adopted as a performance indicator for U K universities. Its capacity to
provide crucial information, at a remarkably low cost, about course quality for funding
agencies, universities, prospective students and employers of graduates is a particularly
attractive characteristic.

Acknowledgements
We would like gratefully to acknowledge the Graduate Careers Council of Australia for
providing some of the data used in the study, and Roland Simons for his patient and
persistent analysis.

Correspondence: Keithia Wilson, School of Applied Psychology, Faculty of Health and Be-
havioural Sciences, Griffith University, Nathan, Queensland 4111, Australia.

REFERENCES
AINLEY,J. & LONG,M. (1994) The Course Experience Survey: 1992 graduates (Canberra, Australian Govem-
ment Publishing Service).
ASHENDEN,D. & MILLIGAN,S. (t 995) Good Universities Guide 1996 to Australian Universities (Victoria, Reed
Reference).
AUSTRALIANVICE CHANCELLORS'COMMITTEE~STANDING COMMYI~'EEON STATISTICS (1995) A review of the
Graduate Careers Council of Australia (GCCA) Graduate Destination Survey and Course Experience Question-
naire (Canberra, AVCC).
BARRETt,P.T. & KLINE,P. (1981) The observation to variable ratio in factor analysis, Personality Study &
Group Behaviour, 1, pp. 23-33.
 Validation of the Course Experience Questionnaire 51

BATE, F., BARRE~t~r,J. & MOIR, F. (1995) Results of the Course Experience Questionnaire: 1993 graduates
(Adelaide, Murdoch University, Academic Services Unit and the Planning Section).
BENTLER, P.M. (1989) EQS: structural equationsprogram manual (Los Angeles, BMDP Statistical Software).
BENTLER, P.M. & BONETr, D.G. (1980) Significance tests and goodness of fit in the analysis of covariance
structures, PsychologicalBulletin, 88, pp. 588-606.
BERNSTEIN,I.H. & TUNG, G. (1989) Factoring items and factoring scales are different: spurious evidence for
multidimensionality due to item categorization, PsychologicalBulletin, 105, pp. 467-477.
BIGGS, J.B. (1979) Individual differences in study process and the quality of learning outcomes, Higher
Education, 8, pp. 381-394.
BIGGS, J.B. (1985) The role of metalearning in study processes, British ffournal of Educational Psychology, 55,
pp. 185-212.
CATTELL,R.B. (1965) Higher order factor structures and reticular-vs-hierarchical formulae for their interpret-
ation, in: C. BANKS& P . L BROADHURST(Eds) Studies in Psychology Presented to Cyril Butt, pp. 223-266
(London, University of London Press).
CATTELL, R.B. (1966) The scree test for the number of factors, Multivariate Behavioral Research, 1, pp.
245-276.
COLE, D.A. (1987) Utility of confirmatory factor analysis in test validation research, Journal of Consulting &
Clinical Psychology, 55, pp. 584-594.
COMI~V, A.L. (1973) A First Course in Facwr Analysis (New York, Academic Press).
CRONBACH,L.J. (1951) Coefficient alpha and the internal structure of tests, Psychometrika, 35, pp. 297-334.
ELEY, M.G. (1992) Differential adoption of study approaches within individual students, Higher Education, 23,
pp. 231-254.
ENTWISTLE,N.J. & KOZEIa, B. (1985) Relationships between school motivation, approaches for studying and
attainment among British and Hungarian adolescents, British Journal of Educational Psychology, 55, pp.
124-137.
ENTWISTLE,N.J. & RAMSDEN,P. (1983) Understanding Student Learning (London, Croom Helm).
EN~rWISTLE, N.j'. & TArt, H. (1990) Approaches to learning, evaluations of teaching and preferences for
contrasting academic environments, Higher Education, 19, pp. 169-194.
EN~XnSX~E,N J., H a n d y , M. & HotnqSEta, D. (1979) Identifying distinctive approaches to studying, Higher
Education, 8, pp. 365-380.
GIBBS, G. & LUCAS,L. (1995) Using research to improve student learning in large classes, paper presented
at the 3rd International Improving Student Learning Symposium, Exeter (mimeo).
GUADAGNOLI,E. & VELICER,W.F. (1988) Relation of sample size to the stability of component patterns,
Psychological Bulletin, 103, pp. 265-275.
HIGHER EDUCATIONCOUNCIL (1990) Higher Education: the challenges ahead (Canberra, Australian Govern-
ment Publishing Service).
HIGHER EDUCATION COUNCIL~NBEET (1992) Higher Education: achieving quality (Canberra, Australian
Government Publishing Service).
KAISER, H.F. (1974) An index of factorial simplicity, Psychometrika, 39, pp. 31-36.
KEMBER, D. & Gow, L (1989) A model of student approaches to learning encompassing ways to influence
and change approaches~ Instructional Science, 18, pp. 263-288.
KERLINGER, F.N. & PEDHAZUR,E.J. (1973) Multiple Regression in Behavioral Research (New York, Holt,
Rinehart, & Winston).
Ln,r ~ , R.D. (1991) Report of the Research Group on PeoCormanceIndicators in Higher Education (Canberra,
Australian Government Publishing Service).
McIr,~Is, C. (1995) Enhancing the first year experience, report prepared by the Centre for the Study of
Higher Education, University of Melbourne (mimeo).
MACNAIR,G. (1990) The British Enterprise in Higher Education Initiative, Higher Education Management, 2,
pp. 60-71.
MA~H, H.W, (1987) Students' evaluations of university teaching: research findings, methodological issues,
and directions for future research, InternationalJournal of Educational Research, t 1, pp. 255-388.
MAgTON, F. & SALJO,R. (1976) On qualitative differences in learning II--Outcome as a function of the
learner's conception of the task, British Journal of Educational Psychology, 46, pp. 115-127.
MEYER, J.H.F. & PARSONS,P. (1989) Approaches to studying and course perceptions using the Lancaster
Inventory, Studies in Higher Education, 14, pp. 137-155.
NORUSIS, M. (1993) SPSS for Windows: professional statistics release 6.0 (Chicago, IL, SPSS Inc.).
PEDHAZUR,E.J. & PEDHAZURSCHMELKIN,L. (1991) Measurement, Design and Analysis: an integrated approach
(London, Lawrence Erlbaum Associates).
52 K.L. Wilson et al.

RAMSDEN,P. (1991) A performance indicator of teaching quality in higher education: the course experience
questionnaire, Studies in Higher Education, 16, pp. 129-150.
RAMSDEN,P. (1992) Learning to Teach in Higher Education (London, Roufledge).
RAMSDEN,P. (1995) Analysis of the Student Opinion Survey for Griffith University, unpublished manuscript.
RAMSDEN,P. & ENTWISTLE,N.J. (1981) Effects of academic departments on students' approaches to studying,
British Journal of Educational Psychology, 51, pp. 368-383.
RAMSDEN,P., MARTIN, E. ~: BOWDEN,J. (1989) School environment and sixth form pupils' approaches to
learning, British Journal of Educational Psychology, 59, 129-142.
PdCHARDSON,J.T.E. (1994) A British evaluation of the course experience questionnaire, Studies in Higher
Education, 19, pp. 59-68.
SARIS,W.E. & STRONKHORST,L H . (1984) Causal Modelling in Non-experimental Research: an introduction to the
LISREL approach (Amsterdam, The Netherlands Sociometric Research Foundation).
TABACHNICK,B.G. & FIDELL, L S . (1989) Using Multivariate Statistics, 2nd edn (New York, Harper CoUins).
TRIGWELL, K. & PROSSER,M. (1991) Improving the quality of student learning: the influence of learning
context and student approaches to learning on learning outcomes, Higher Education, 22, pp. 251-266.
VELICER~W.F., PEACOCK,A.C, • JACKSON,D.N. (1982) A comparison of component and factor patterns: a
Monte Carlo approach, Multivariate Behavioural Research, 17, pp. 293-304.
WELL, S. (1992) Creating capability for change in higher education: the RSA initiative, in: R. BARtlETt (Ed.)
Learning to Effect, pp. 186-203 (Milton Keynes, Society for Research into Higher Education & Open
University Press),.
WHEATON, B. (1987) Assessment of fit in overidentified models with latent variables, Sociological Methods and
Research, 16, pp. 118-I54.
YORKE, M. (1995) Siamese twins? Performance indicators in the service of accountability and enhancement,
Quality in Higher Education, 1, pp. 13-30.
 V a l i d a t i o n o f the Course E x p e r i e n c e Q u e s t i o n n a i r e 53

Appendix. The Course Experience Questionnaire (CEQ36, CEQ30, CEQ23)

Instructions

In answering this questionnaire, please think about the course as a whole rather t h a n identifying individual
subjects, topics or lecturers. T h e questions relate to general issues about your course, based on c o m m e n t s that
students have often m a d e about their experiences of university teaching and studying. Your responses are
strictly confidential a n d will n o t be seen by teaching staff.

Scoring

Items are scored on a scale from 1 to 5, where 1 m e a n s 'definitely disagree' and 5 m e a n s 'definitely agree',
save for those printed in italics, which are scored in the opposite direction.
to u n d e r s t a n d difficulties students m a y be having with their work AW
Items
* 1 It's always easy here to know the standard of work expected CG
* 2 T h i s course has helped m e to develop m y problem-solving skills GS
# 3 T h e r e are few opportunities to choose the particular areas you want to study IN
# * 4 T h e teaching staff of this course motivate students to do their best work GT
#* 5 The workload is too heavy AW
* 6 This course has sharpened m y analytic skills GS
# 7 Lecturers here frequently give the impression they have nothing to learn from students AA
# * 8 You usually have a clear idea of where you're going and what's expected of you CG
# * 9 Staff here p u t a lot of time into c o m m e n t i n g on students' work GT
# * 1 0 To do well on this course all you really need is a good memory AA
*11 This course has helped develop m y ability to work as a team m e m b e r GS
"12 As a result of doing this course, 1 feel more confident about tackling unfamiliar problems GS
"13 This course has improved m y written c o m m u n i c a t i o n skills GS
# 14 It seems to me that the syllabus tries to cover too many topics AW
# 15 T h e course has encouraged m e to develop m y own academic interests as far as possible IN
# 16 Students have a great deal of choice over h o w they are going to learn in this course IN
# ' 1 7 Staff seem more interested in testing what you've memorised than what you've understood AA
# ' 1 8 It's often hard to discover what's expected of you in this course CG
# ' 1 9 W e are generally given e n o u g h time to u n d e r s t a n d the things we have to learn AW
# * 2 0 T h e staff make a real effort to u n d e r s t a n d difficulties students m a y be having with their work G T
# 21 Students here are given a lot of choice in the work they have to do IN
# * 2 2 T e a c h i n g staff here normally give helpful feedback on how you are going GT
# * 2 3 O u r lecturers are extremely good at explaining things to us GT
# 24 The aims and objectives of this course are N O T made very clear CG
# * 2 5 T e a c h i n g staff here work hard to make subjects interesting GT
# * 2 6 Too many staff ask us questions just about facts AA
# * 2 7 There's a lot of pressure on you as a student here AW
*28 This course has helped me develop the ability to plan m y own work GS
# 29 Feedback on student work is usually provided O N L Y in the form of marks and grades AA
# 30 W e often discuss with our lecturers or tutors how we are going to learn in this course IN
# 31 Staff hem show no real interest in what students have to say GT
# 32 It would be possible to get through this course just by working hard around exam times AA
# 33 T h i s course really tries to get the best out of all its students GT
# 34 There's very little choice in this course in the ways you are assessed IN
#*35 T h e staff here make it clear right from the start what they expect from students CG
# * 3 6 The sheer volume of work to be got through in this course means you can't comprehend it all AW
thoroughly
37 Overall, I a m satisfied with the quality of this course
Note: G T = G o o d T e a c h i n g scale; C G = Clear Goals a n d Standards scale; GS = Generic Skills scale;
A A = Appropriate A s s e s s m e n t scale; A \ V = Appropriate Workload scale; I N = Emphasis on I n d e p e n d e n c e
scale. Items 1 - 3 6 = C E Q 3 6 ; # items=CEQ30 (used in R a m s d e n , 1991; Richardson, 1994);
*items = C E Q 2 3 .