To assess if SET forms are effective in measuring the factors related to effective teaching we need something to compare them to.
To find what a viable option could be, the first step is to define exactly what student evaluations of teaching aim to measure. Luckily for us, Elizabeth Barre, Co-Chair of the Committee on Teaching and Course Evaluations from Rice University, has conducted a thorough literature review of 68 peer-reviewed publications. Among many interesting findings, Elizabeth has audited what student evaluations actually aim to measure and compiled the conclusions into three categories. Generally, what SETs and course assessments measures are:
- Student Attitudes — What students think about the course and/or the instructor.
- Teaching Practices — What tactics the teacher are using to transfer knowledge?
- Student Achievement — What the outcome from the course is.
Together, these three areas can be combined to form a reasonably holistic view on how effective the teacher is. Assuming we’re able to measure these areas reliably and accurately, that is.
So, if we’re not using student surveys, how can we go about to measure these topics?
Well, starting with Student Attitudes, the most obvious way to measure this in another way is by simply talking to the students. Qualitative methods such as personal interviews or focus groups are proven to produce a deep understanding of underlying reasons and motivators. In market research, it’s often common to start off with a qualitative study of the subject, base a hypothesis from the results and use quantitative research, such as surveys, to catch larger trends. A conversational approach to feedback is also useful since it’s not as structured as a survey inherently is. An interview can include follow-up questions, additional insights from body language and a content customisable to respondent answers.
Alternatively, alumni surveys is also a plausible option when it comes to measuring student attitudes. A common perception is that students might not appreciate hard work until years later when hard work has paid off.
Cool. So we have some alternatives to measure the first category by. How do these compare to regular SET-forms then? In the literature study, Barre looked into that. Here is what she found:
On the Pearson scale (where +1 is a perfect correlation and -1 is a perfect inverse correlation), student evaluations of teaching surveys correlate with qualitative responses by between 0.75-0.93 .
Alumni surveys aren’t far behind with a correlation of between 0.69–0.83. Both are very strong correlations.
This means we can, in most cases, expect a similar result from SETs, qualitative responses, and alumni surveys. We might not get as detailed replies as in an interview or the retrospective analysis of an alumni survey. BUT, why aren’t these methods used more widely then?
Qualitative responses would most likely produce more qualitative answers, as the name implies. However, the problem is of course that they have to be conducted and analysed, and that take time. A LOT of time. And since time equals money, few schools regularly conduct student interviews.
Alumni surveys then;
Sure, it’s a good thing to get the retrospective angle, but it’s not optimal for making effective improvement to wait a few years before asking for feedback. Detailed information is also close to hopeless to remember for most due to memory bias which is likely to skew results.
Moving on to Teaching Practices. To find out how the instructor acts to get the message across is an important factor in the overall effectiveness. Other than asking students to fill out a survey there’s generally a few things that could be tested. A commonly used tactic is to have a senior faculty member, a principal or a teaching expert observe classroom behaviour. For a delayed observation, video recordings can also be used.
Another approach is for the teacher to simply-self assess practices.
Barre found strong correlations between student evaluations of teaching forms and expert observations (0.50–0.76), peer evaluations (0.48–0.69) and administrator evaluations (0.39–0.62) and moderate correlations to self-assessments (0.29–0.49).
Having someone who is well known to effectively transfer knowledge observe and evaluate your performance and give you feedback should be a sure way of getting accurate data and points where improvements can be made. This alternative is a constantly returning option proposed by scholars who studied teacher feedback, and it sounds really reasonable. Which it is. But still, caution is needed when analysing results. Educators could choose acquainted colleagues to assess them and peers’ might be less of a pedagogical talent than the one being assessed in the first place. After all, in higher education, the main focus of most professors lie in the narrow research field and not on pedagogy.
Furthermore, the experience of having a supervisor assess your performance can often be an uncomfortable experience with, at least, the same magnitude of stress involved in reading student evaluations.
Self-evaluations, still with a moderate correlation to student surveys, can surely work well as a complement to surveys or other methods. Evaluating teachers based solely on self-assessments might not be the best of ideas for obvious reasons, however.
How about Student Achievement? If we’re assuming that teacher effectiveness influences student performance, measuring it is a strong alternative to SETs. Measures can be attained by studying either exam results or grades to find out how much knowledge students actually gain from their classroom experience.
Caution is, however, necessary when comparing teacher effectiveness and test scores/grades. Since, as we saw in the last post, high scores can reflect a teacher who is generous in his grading or giving easy exams.
Correlations between student evaluations and student evaluations sum up to 0.39–0.50(strong correlation) for exam scores and 0.10–0.30(weak correlation) for grades. We will discuss grades more in depth in a coming article.
Barre was even kind enough to provide us a perceptive graph, based on this meta-analysis study by Peter Cohen. It’s showing the number of studies that examined the correlation between student ratings and achievement.
As made clear by the graph; sure, there are studies that found negative correlation and close to no correlation whatsoever. But more importantly: there are many, many more studies carried out that found positive and very positive correlations. In total, there are 10 studies showing a negative correlation compared to 79 showing positive correlations. This graph must be considered a pretty decisive overview on what the research community have exposed over the years.
Another thing worth mentioning is that this graph shows that a greater deal of student achievement can be attributed to the individual instructor rather than the course itself.
Replacing SETs with any of these methods is most likely to produce roughly the same results. Out of the discussed alternatives, there simply aren’t a better solution to collecting student opinions.
So, the answer to the question ‘Why aren’t we using another way to collect teacher feedback?’ is simply:
Student evaluations of teaching forms are the best compromise between insights, practicality, and workload. In other words; It’s the best tool available for now.
In the next section, we’ll look into what bias can affect your evaluation results when utilising SETs.