Do student evaluations measure teaching effectiveness? [Part 3 – Evaluation as symbol]]

By | March 31, 2014

I believe that it is important to think about why these student course evaluation ratings are being obtained, and how that relates to the kinds of measures used. I pointed out in Part 2 that the one-shot end-of-course retrospective rating system cannot obtain ratings valid over the whole term. There are procedures that could obtain valid ratings, but they are a good bit more complicated.

Most usually, the one-shot ratings are mandated by administrators, some of whom probably believe that they are valid (not being educated about such things) but many of whom must understand their limitations. So why do they continue to mandate the use of something that they know is not a valid measure of what it purports to measure?

I believe that a large element of the answer to this question goes back to one of my favorite political scientists, Murray Edelman, and his enormously valuable 1964 book, The Symbolic Uses of Politics. Here and elsewhere, he developed the thesis that politics is increasingly dominated by symbolism over substance. As his 2001 obituary in the New York Times put it:

“He believed that public political developments often act as what he called symbols with meanings or purposes that can be determined only by careful investigation. ‘A fact is always embedded in a theory and has to be interpreted,’ he once said…. he saw the symbolic value of democratic elections as their ability to convince voters that they face more of a political choice than is often, in fact, the case. Similarly, he regarded the elaborate rituals of the courts as intending to inspire awe and add authority to their judgments.”

Although he wrote nearly 50 years ago, most if not all of his analysis remains fresh and current – perhaps even more so than originally. Then, it could be argued that many of those engaged in the manipulation of political symbols weren’t actually all that conscious of what they were doing. Fifty years later, we have a whole grandchild cohort of political operatives quite shamelessly doing the same thing, this time fully aware that they are engaged in shows of political theater and proud of it.

It’s become clear in the intervening years that his ideas applied not only to national politics but to the micro-worlds of organizational politics as well. Organizations engage in the manipulation of symbols in precisely the same way as do governments, and with many of the same consequences. Universities manipulate symbols of effectiveness and accountability in pretty much the same ways that political ideologues and operatives wave flags and ritually denounce traitors. And the result is the same kind of debasement of the dialogue around effective teaching that has characterized the collapse of the national political narrative into largely a set of gibbering talking points.

Thus, the question of student evaluations must be seen as political as well as pedagogical. Specifically, I contend that much of the interest of university administrators in student evaluations of classes lies in their symbolic value – that is, they give the impression that the university cares about student input and opinions. The one-shot post-class survey is the easiest possible way to generate output that’s somewhat face-credible and that they can wave around if ever questioned. The administrators may not know or they may not know and simply not care that the results obtained from these surveys usually generate numbers that have no valid interpretation – the fact that they generate numbers with symbolic value is the important point.

A secondary benefit is that the same numbers can also be used to intimidate faculty and others. In accordance with Di Franco’s Tool Principle, they can wave the ratings in the face of any faculty member over whom they need an advantage, for whatever purposes. Since the symbol has been established, then any response that might question the value of the ratings must first overcome the symbol itself, which is almost impossible. Note that this intimidation value of the ratings need not be actively employed. The fact that it exists as a symbol practically guarantees that anyone who participates in that symbol system will understand that it might be so employed, and will undertake to intimidate themselves proactively.

I strongly believe in the need to assess student reactions to classes, and to obtain useful feedback about what’s working and what isn’t, in time to make appropriate changes.  In the FTF environment, observable qualitative cues (student body language, class behavior, questions, etc.) may themselves provide the vast bulk of evaluative information (assuming that the instructor is attuned to them). But it’s even more essential in online classes, where most of the FTF evaluation mechanisms are unavailable. Most LMSs do not provide for anything like real-time student feedback; any instructor who is really interested in this will probably need to devise additions to the LMS to obtain it.

Any system that waits for a retrospective end-of-class survey to provide student feedback, in either FTF or online environments, is generally a sign that the evaluation is more symbolic than substantive and political rather than pedagogical. The more that we ignore or discount the symbolic value of evaluation, the less likely we are to ever get an evaluation that works in real rather than purely symbolic terms.