What Program Directors can do to improve CEQ scores
- Introduction
- What is the CEQ?
- What does the CEQ measure?
- What is the relationship between the CEQ, CEI and SET?
- What types of CEQ data are available?
- How can I evaluate the quality of the available CEQ data?
- References
- Implications of CEQ feedback for program direction
Introduction
The quality of teaching at UniSA is highly valued because it results in better outcomes for our graduates. Measuring this quality is a complex process that takes place at the course, program, university, state and national levels. The processes involve multiple stakeholders, including students’ opinion of their experiences while studying at university.
The purpose of this guide is to:
- provide Program Directors with an overview of the CEQ, what it measures and how it relates to the Course Evaluation Instrument (CEI) and the Student Evaluation of Teaching (SET)
- assist Program Directors in evaluating CEQ feedback
- suggestions to improve student and graduate outcomes.
What is the CEQ?
The Course Experience Questionnaire (CEQ) is a national survey of teaching quality that commenced in 1993. Each year graduates from the previous year at all universities across Australia are invited to contribute to a questionnaire designed by the Graduate Careers Australia (GCA) about their experience in their program (CEQ) and their subsequent employment outcome (Graduate Destination Survey - GDS) . The survey is administered by Planning and Assurance Services for UniSA and collated by GCA.
Currently the CEQ portion of the questionnaire (PDF 95.7kb - download Adobe Acrobat) consists of 24 items that are grouped to produce learning experience scales measuring clear goals, good teaching, appropriate workload, generic skills, appropriate assessment and overall satisfaction.
While there are many debates about the validity of the CEQ, the reality is that the data gathered is used to rank universities across Australia in the Good Universities Guide. The analysis is very public and is promoted in various ways including the national and local media and in the marketing of universities. Consequently, taking an active and positive interest in the CEQ and its results are very important.
The University’s CEQ feedback is made available annually through Planning and Assurance Services. Also available in UniSAinfo Reporting are CEQ data from other universities with similar program profiles to ours. This allows Program Directors at UniSA to benchmark their program’s performance against others in comparable Field of Education (FOE).
What does the CEQ measure?
The CEQ is based on research conducted by Ramsden (1991). He found that students, who took a ‘deep’ approach to their learning, learned more effectively and gave positive responses to the CEQ items. In contrast students who took ‘surface’ approaches to learning consistently gave negative responses to CEQ items.
The question framework of the CEQ is based around:
- students are clear about the standard of work expected, know where they are going and what is expected of them, and feel that staff clearly communicate their expectations. Clear Goals Scale
- students get useful qualitative feedback on their progress, and find that their lecturers explain things effectively, make the program interesting, motivate students, and understand students’ problems. Good Teaching Scale
- students find the workload is manageable and allows them enough time to understand what they are learning. Appropriate Workload Scale
- students report that the program has helped them to develop problem-solving skills, including confidence with unfamiliar problems; analytic and communication skills; the ability to work as a team member; and the ability to plan their own work. Generic Skills Scale
- assessment focuses on understanding, not just recall of information. Appropriate Assessment Scale
- other factors that impact on the learning experience are optimized. Overall Satisfaction.
What is the relationship between the CEQ, CEI and SET?
One of the main drawbacks of the CEQ is the time delay in the feedback. For example if a student was studying a 4 year program from 2001 to 2004; they would graduate and be surveyed for the CEQ in 2005. This data would become available in 2006.
To provide Program Directors and Course Coordinators at UniSA with timely feedback , the Course Evaluation Instrument (CEI) and Student Evaluation of Teaching (SET) surveys were developed with items that have relationships to the CEQ items. The aim is to provide meaningful and timely insights into students’ current teaching and learning experiences.
What types of CEQ data are available?
Current CEQ data is available through UniSAinfo reporting (UniSA Public Folder; section 7.0 Evaluation; 7.1 Course Experience Questionnaire). Results are progressively received and made available to staff each year. Typically, in November of the survey year we are able to access our own data, and in February of the subsequent year, national comparative data for benchmarking becomes available. PowerPlay Cubes are dynamic tables which, with training, you will be able to use to make sophisticated analysis of data. In PowerPlay web reports, the user makes selections to indicate a program, and a page of information is generated for printing.
Contact the Planning and Assurance Services web site for support accessing this data.
How can I evaluate the quality of the available CEQ data?
Student feedback from CEQ items needs to be evaluated for quality before it can be can be used to improve teaching and learning. This involves:
- looking at response rates
- examining the mean score and agreement rate for each of the scales and
- benchmarking how the program’s CEQ data compares to data from similar programs in different Universities.
How do response rates effect the data?
The higher the response rate, the more confidence you can have in the generalisations you are drawing from the information. Prior to 2004, if response rates are less than 10 then the CEQ data is not reported, unless it is trend data, then response rates below 5 are omitted. Extreme caution is required when interpreting data from programs with low response rates. If you want to look at strategies to improve response rates then please go to the resource on the left hand menu.
What is a mean score?
The mean score has been determined from the Likert data for each of the Learning Experience scales. The data from each of the CEQ items are collated into strongly agree, agree, neutral, disagree, strongly disagree and not applicable (no response) categories. Each response is converted to its numerical equivalent (SA=+100; A=+50; N=0; D=-50; SD=-100), totalled, and then divided by the number of responses to obtain a mean or average. A positive mean score indicates that more respondents agree with the scale than disagree. A negative mean score indicates that more respondents disagree with the scale than agree.
What is the agreement rate?
The agreement rate is useful for validating the strength of the data. The larger the agreement rate the more confident you can be when interpreting the mean score data. Care is required interpreting CEQ data that has a low agreement rate (indicating a significant percentage of respondents indicated neutral/undecided in response to the item).
How can I analyse CEQ free text responses?
Within UniSAinfo Reporting can be found CEQuery -a software program designed by Graduate Careers Australia to assist higher education in its analysis of graduates’ qualitative responses to the Course Experience Questionnaire (CEQ). The cube represents the frequency count of the ‘hits’ recorded against the dictionary for both ‘best aspects’ and ‘needs improvement’ comments collected in the CEQ. The cube can assist users to determine trends in graduates’ comments or to identify particular areas that record unusually high or low frequencies. To access this data contact Planning and Assurance Services.
How can I benchmark the program to other similar programs?
Students in some disciplines seem to be consistently more critical than in others. It has been argued that for this reason, the absolute CEQ score is only helpful when it is compared with CEQ data from groups of students from similar fields of study. For example, the CEQ of a Bachelor of Commerce program needs to be compared with other Bachelor of Commerce programs, not with the CEQ of, say, a Social Work program. By using the CEQ benchmarking tables it is possible to benchmark like against like, using either the national average, or a direct comparison to select institutions.
References
Ramsden, P 1991, ‘A performance indicator of teaching quality in higher education: The course experience questionnaire’, Studies in Higher Education, vol. 16:, pp. 129-150.
Ramsden, P 1998, ‘Learning to lead in higher education’ Routledge, London..
Implications of CEQ feedback for program direction
Choose one of the CEQ scales below to explore possible program directions. For each scale there is a description, a set of key questions for program directors and program teams, and some suggestions for change.
- Responding to the Clear Goals and Standards Scale
- Responding to the Good Teaching Scale
- Responding to the Appropriate workload Scale
- Responding to the Generic skills Scale
- Responding to the Appropriate Assessment Scale
- Responding to the Overall Satisfaction item
Responding to the Clear Goals and Standard Scale
Description
This scale measures whether students perceive the aims, objectives and assessment processes of the course are expressed adequately. It determines whether students perceive a clear and consistent message about the requirements for success in the program.
Key questions for Program directors and program teams
- Which courses are doing well in this scale, and which are doing poorly?
- Do you have a set of all the current course information booklets? Are course information booklets uniformly set out across all courses? Do each have clearly expressed aims and objectives for the course? Are they presented similarly online?
- Does the Program team, including sessional and adjunct staff, know and understand how the graduate qualities are developed in your discipline? Do they share a common language about the graduate attributes? Do they use that language when communicating to the students?
- Do the course information guides use the graduate quality language consistently?
- Does each assessment task have clearly stated aims and objectives?
- Are students given marking guides or criteria for every assessment item? Is the development of graduate qualities through assessment tasks a clear part of the assessment criteria?
- Can you guarantee that all students get the same information across and within courses?
Suggestions for change
- Using the Program Director function of the CEI, analyse the student responses to item 1 across the courses. Is it as low as that indicated by the CEQ? If so, determine those courses that have negative scores for this item, or the courses that are also low positive. Determine those coordinators who have scored strongly in this item. Use focus groups of students from final year courses to determine which areas of the curriculum are performing poorly and performing strongly in the expression of aims and objectives. Does this information relate well to the CEI data? Consider setting up mentoring relationships between staff who score strongly and weakly in item 1 of the CEI. Allow them time to work together to develop clear aims, objectives and marking criteria to share with students prior to doing the assignment.
- Consider publishing a program information booklet that collates resources for all the courses. Review this resource for consistency and clarity.
- Use the LTU/Division service agreement to engage professional development staff from Learning and Teaching Unit to mentor staff through the process of articulating clear aims, objectives and marking criteria in light of the graduate qualities. Similarly, the service agreement can be used to engage learning adviser staff to work with staff to create online workshops to support students during assessment tasks.
- Use Program team meetings to discuss the Graduate Qualities and their indicators and agree on which Graduate Quality indicators are appropriate for graduates from your discipline.
- Consider agreeing on a consistent online presence for at least course information available online, so that students develop an understanding of where to go to access information they need about what is expected of them.
- Utilise online discussion forums for each assessment item where students can lodge all questions, queries and other communication about assessment tasks with the course co-ordinator. This guarantees that all students (and teaching staff) can access a clear and consistent message about expectations associated with assessment tasks.
- Consider making it a policy that marking criteria/guides are provided to students before they do the assignment.
Responding to the Good Teaching Scale
Description
This scale measures whether graduates perceived that they received useful qualitative feedback on their progress, and found that staff explained things effectively, made the program interesting, motivated students, and attempted to understand students’ problems. It is very much about the relationship between the teacher (course coordinator and/or tutor) and the learner.
Key questions for Program directors and program teams
- Which courses are doing well in this scale, and which are doing poorly? Which elements of the scale are impacting most?
- Is there a reconfiguration of assessment and learning activities in response to student’s recognised prior learning?
- Are students provided with a choice in assignment tasks in all courses in the program?
- Are there sufficient opportunities within assessment tasks to relate the task to the student’s experience and/or background knowledge?
- Are standardised and agreed mechanisms for feedback used in the program?
- Are teachers in the program encouraged to conduct scholarly research about teaching and learning and to discuss and publish their teaching practice?
- Are a variety of teaching and assessment approaches used in the program?
- Are large numbers of part-time staff used to teach/tutor in this program?
Suggestions for change
- Use a CEI item analysis to identify those courses performing strongly and poorly in items 2, 4, 7 and 9. Utilise student focus groups, particularly of final year students, to cross check the information from the CEI item analysis and CEQ data. Consider setting up mentor/peer review relationships between teaching staff in courses displaying strength and weakness in these areas.
- Encourage all staff to use the course information proforma to present clear and consistent course information. Incorporate your team’s common understandings of the graduate qualities into the aims and objectives of the courses and each assessment item. Run professional development sessions about feedback.
- Encourage the use of standardised feedback sheets. Develop marking moderation systems. Use external assessment where appropriate.
- Consider running academic development sessions to support the use of online quizzes with embedded feedback to support self-directed learning.
- Engage sessional and continuing Teaching staff in teaching and learning programs such as ‘Teaching @ UniSA’ or the ‘Graduate Certificate in Higher Education’. Combine these program’s assignments/activities with development required in courses to improve teaching and learning.
- Establish a mechanism for systematic peer review of teaching for both continuing and sessional teaching staff in your program.
- Conduct academic development sessions that allow teachers in the program to share best practice with other teaching staff (including sessional teaching staff).
- Engage your teaching team with the teaching and learning literature related to your discipline.
- Identify reusable learning objects that can be integrate in courses.
- Encourage teaching staff to present their teaching and learning experiences in educational streams at discipline-specific conferences.
- Encourage staff to apply for scholarly teaching awards.
- Work with staff from Program team on presentation skills in lectures and tutorials that encourage students to engage with learning (eg. Case studies, anecdotes, professional applications).
Responding to the Appropriate Workload Scale
Description
This scale measures whether graduates perceive that the workload in the program was manageable and allowed them enough time to understand what they were learning.
Key questions for Program directors and program teams
- Which courses are doing well in this scale, and which are doing poorly?
- Are Course Coordinators linking workload to assessment or are there numerous activities not related to assessment?
- Do all teaching staff understand the relationship between points and student work (as clarified in Policy A-13 Recognition of prior learning) where 1 unit (of a 4.5 unit course) should equate to 40 hours of student effort?
- Have new coordinators, who have moved into teaching topics, added their preferred learning materials to the student load, but have not felt confident enough to remove previous learning materials?
- Have Course Coordinators moved into online teaching without considering an appropriate adjustment of student reading load?
Suggestions for change
- Consider mapping the assessment tasks if this hasn’t already been done. Consider the diversity of assessment tasks and the skill development that is measured in each. Is there repetition?
- Perform a CEI item analysis for item 6 using the Program Director functionality. Identify the courses which students have indicated have a high or low workload. Use focus groups with graduates to confirm which parts of the program were onerous – is it all assessment or are there other aspects that impact on student workload? Share results of focus groups and CEI analysis with staff in Program team meetings. Consider setting up peer mentoring relationships between staff teaching in courses that do well in this area and those that do poorly.
- Consider running an academic development workshop on assessment, emphasising the relationship between learning objectives and assessment. Make a point of emphasising the relationship of the unit weighting of courses to hours of student effort. Professional development staff at Learning and Teaching Unit are available to facilitate these sessions.
- Use the CEQ data to identify programs that perform well in this area. Encourage teaching staff to establish collaborative relationships with other universities teaching similar programs. Sponsor attendance at conferences where the teaching of the discipline is a consistent theme. Encourage discussion of assessment strategies and student workload in these forums.
Responding to the Generic Skills Scale
Description
This scale measures whether graduates perceive that the program has helped them develop problem-solving skills, including confidence with unfamiliar problems; analytic and communication skills; the ability to work as a team member; and the ability to plan their own work.
Key questions for Program directors and program teams
- Which courses are doing well in this scale, and which are doing poorly? Which elements of the scale are impacting most?
- Are employers aware of the way the curriculum in your program is developed?
- Are graduate qualities progressively developed from course to course and year to year?
- Do you have a mechanism for students to track their development of Graduate Qualities to record their achievements and experiences?
Suggestions for change
- Interview employers of your graduates, members of professional associations, graduates, alumni and accreditation bodies. Ask what qualities they want from a graduate employees. Work with industry to integrate their needs in the curriculum. Encourage Course Coordinators to link the professional relevance of the assessment tasks students are being asked to do with the graduate qualities they are developing by completing the task.
- Perform an item analysis of CEI feedback to determine which courses are performing poorly and strongly in item 3 and 8. Conduct focus groups of graduates to tease out which skills they thought they were lacking in their profession. Present findings to the Program team for discussion. Consider establishing peer mentoring relationships between staff teaching in courses that do well in this area and those that do poorly. A teaching guide about responding to CEI feedback is available for Course Coordinators to make positive changes to their teaching.
- Integrate work-based problem scenarios into assessment tasks. Encourage staff to take PEP, or even a short placement within professional environments. Utilise professionals as sessional teachers or guests. Provide opportunities for sessional professionals to share with continuing staff their experiences and how they relate to different courses.
- Map the assessment tasks at each year level. Ensure there is an increasing level of complexity from study period to study period. For example basic problem solving skills should be provided in year 1, with more advanced problem solving skills required in year 2. Alternatively conduct a graduate quality audit of the program to determine where misalignment between stated GQ profiles and the assessment of graduate qualities exist. Present your findings to the Program team for discussion and action.
Responding to the Appropriate assessment Scale
Description
This scale measures whether graduates perceive that assessment focuses on understanding, not just recall of information. Courses that contain too much content force students to adopt surface learning strategies to cope.
Key questions for Program directors and program teams
- Which courses are doing well in this scale, and which are doing poorly? Which elements of the scale are impacting most?
- Do assessment tasks relate to professional contexts? Are tasks authentic and meaningful?
- Do students value and choose to engage in assessment tasks because they understand that what they will learn during that experience will benefit them as a professional?
- Do course materials clarify the value of learning outcomes of assessment tasks?
- Do you use a large number of exams, which mainly assesses the body of knowledge?
Suggestions for change
- Consider mapping the assessment tasks across the program. Is there diversity in the assessment tasks? Conduct a graduate quality audit of the assessment tasks within the program. As assessment is the only way we can guarantee that GQs are indeed developed, are all the GQs promised within the program profile evidenced in assessment? Does assessment in the program focus on Graduate quality 1, the body of knowledge?
- Perform an item analysis of CEI feedback to determine which courses are performing poorly and strongly in item 8 and 5. Conduct focus groups of graduates to tease out which courses are promoting surface learning strategies. Present findings to the Program team for discussion. Consider establishing peer mentoring relationships between staff teaching in courses that do well in this area and those that do poorly.
- Interview employers of your graduates, members of professional associations, graduates, alumni and accreditation bodies. Ask what they require from graduate employees. Integrate authentic problem scenarios into assessment tasks. Build in projects, professional placements, simulations, particularly in the final years of the program, that provide experiences that prepare graduates for professional life.
- Encourage staff to take PEP, or even a short placement within professional environments. Utilise professionals as sessional teachers. Provide opportunities for sessional professionals to share with continuing staff their experiences and how they relate to different courses.
Responding to the Overall Satisfaction item
Description
This item measures whether graduates perceive that other factors that impact on the learning experience have been optimised
Key questions for Program directors and program teams
- Does your program have its own culture?
- Is the program connected to a professional body? Are students encouraged to be members?
- Are there any social rituals associated with the program (dinners, balls, barbeques, breakfasts, student publications, quiz nights, awards etc.)?
- Do students have an identity within the program such as a student group or association? Is the group active? Do staff members support this group by attending student functions?
- Do teaching staff attend graduations?
- Are you available for students to discuss ways to improve their graduate outcomes?
- Do the graduates of the program (alumni) continue to have a role to play in the program?
- Do students know about the CEQ before they graduate?
Suggestions for change
- This item can be interpreted in many ways and may be related to some of the other scales. If there is little problem in other areas, and this item alone provides negative feedback then a culture change in the learning within the program may be required.
- Building a culture for a program is achieved by investing time in
building relationships. Students need to feel connected with the program as
a whole, as well as to the School, the University and the profession. As
Program Director, your role is to liaise with students, colleagues and
professional bodies to make this environment as rich as possible. There are
many ways that collectively contribute to building this culture, examples
include:
- organise for year level group photographs to be taken of the students and display in prominent positions in School offices, walls of corridors etc.
- encourage staff to get to know their students, to talk with them informally between classes, and to interact
- foster the formation of student groups. Invest time annually to assist this process
- meet regularly with representatives from student groups to listen to their concerns and suggestions
- support planning of student functions
- develop job boards that alert students to jobs available for graduates
- sponsor some student events (barbeques, dinners, breakfasts, quiz nights) encourage as many staff as possible to attend student functions
- build web sites that celebrate student life, excellence and professionalism
- conduct award presentations that mark excellent student and teaching achievements
- talk to students about the CEQ survey and its importance
- ensure all staff attend and celebrate graduation day – take photographs for web sites and meet parents, partners and friend of graduates
- invite professional societies to sponsor student activities to boost membership
- build exchange programs that allow students to study in different countries
- encourage alumni involvement in the program (talks about professional life and the skills required)