The letters in coloured circles problem: Student evaluation lets us improve, but only the things we can change

Five faces ranging from smiling to frowning, representing a scale

I found this post about student evaluations deeply affecting (CW: self-harm, general realness). The author’s experience resonated.

Every class, as we waited to start, I would ask my students (blue squares on the video conference screen) if they had any gossip. “I need something,” I said. “Any catfights? Anyone get really drunk and moon their neighbors? Anyone’s roommate cheat on their boyfriend? I am desperate here.”

Sometimes there would be a “no :(” in the chat.

Usually just silence.

Response to Student Evaluations, Sarah E. Smith

Online meetings and tutorials where nobody else turns on their screens have been the leitmotif of the past eighteen months. Online discussions that end up becoming discussions only between the teaching assistant and I, while some letters in coloured circles look on.

It’s not dehumanising exactly. It’s… nothing?

Why don’t they turn on their cameras? I don’t know. They might find it emotionally draining to meet via Zoom constantly. They might have a crappy internet connection. They might be doing other things. They might be self-conscious.

It’s not just students. Most meetings with staff are the same. A shared screen and a sea of letters in circles.

Connection seems to be missing. It’s not the medium, and it’s not solely the COVID pandemic. It’s something else.

Elaina Nguyen from Canada and fellow students from Australia, Hong Kong and Malaysia have argued that the values underpinning effective learning partnerships – respect, reciprocity, shared responsibility, authenticity, honesty, responsibility, inclusivity, reciprocity, empowerment, trust, courage, and plurality – all need to renegotiated not just for virtual environments, but also for our COVID world . They’re onto something.

Reading Sarah E. Smith’s post reminded me that some universities in Australia plan to publish subject evaluation metrics. There are proposals that students should be able to see aggregated subject evaluations, which are usually mean agreement scores with statements along the lines of “this subject enabled me to develop an understanding of the subject matter” or “this subject made me feel like part of a community”. Fair enough questions and useful information for teachers.

Student evaluation is unequivocally a good thing. It one of the main ways we can learn about what courses are like for students, and improve the way we teach.

The challenge is that often it’s not solely the teaching, the assessment activities or learning materials that students are evaluating.

Jenna Price captured the challenge at the heart of this in an article yesterday:

When students fill out those forms, they don’t answer the questions before them. La Trobe University lecturer Troy Heffernan has published widely on what students really mean when they rate subjects and the teachers who teach them. It’s about gender, race, accent. It’s also about institutional practices.

What goes wrong when uni students mark their teachers

Evaluation is only helpful if we’re able to act on the insights it offers. If students are using them to express dissatisfaction with student-to-teacher ratios, institutional policies, or have unacknowledged gender or racial biases , their usefulness evaporates. Similarly, evaluation scales become less useful for ongoing improvement if used as proxy metrics for poorly defined staff performance outcomes.

The way forward is clearly to develop more collaborative pedagogies based on students as partners , including co-creation of subjects and recognising students as consultants in that process.

At this time, what we need to improve as teachers probably isn’t captured through an agreement scale, valuable as that could be in the proper context. It’s solving the letters in coloured circles problem. It’s figuring out what’s really going on for the humans behind the circles.

I’d like to add one question to the student evaluation forms: why don’t they turn on their cameras?

References

Ntem, A., Nguyen, E., Rafferty, C., Kwan, C., & Benlahcene, A. (2020). Students as partners in crisis? Student co-editors’ perspectives on COVID-19, values, and the shift to virtual spaces. International Journal for Students as Partners, 4(2), 1–8. https://doi.org/10.15173/ijsap.v4i2.4432
Mercer-Mapstone, L., Dvorakova, S. L., Matthews, K. E., Abbot, S., Cheng, B., Felten, P., Knorr, K., Marquis, E., Shammas, R., & Swaim, K. (2017). A Systematic Literature Review of Students as Partners in Higher Education. International Journal for Students as Partners, 1(1). https://doi.org/10.15173/ijsap.v1i1.3119
Heffernan, T. (2021). Sexism, racism, prejudice, and bias: a literature review and synthesis of research surrounding student evaluations of courses and teaching. Assessment & Evaluation in Higher Education, 0(0), 1–11. https://doi.org/10.1080/02602938.2021.1888075

Ethical Protocol for evaluation in Aboriginal and Torres Strait Islander settings

The ethical protocol provides principles and guidance on how to respect the elders, cultural knowledge, and lands and seas of Aboriginal and Torres Strait Islander Peoples. It provides a tool to frame and design an ethical approach to apply throughout all stages of monitoring and evaluation tasks and processes.
— Read on www.betterevaluation.org/en/themes/indigenous_evaluation/ethical_protocol

A guide to monitoring and evaluating policy influence

A guide to monitoring and evaluating policy influence

A useful and, more importantly, comprehensible guide to aligning types evaluation activities with the anticipated mechanisms of policy influence:

  • evidence and advice
  • public campaigns and advocacy
  • lobbying and negotiation.

There’s also some recognition of the messiness of attribution in the context of policy change, and that even policy actors themselves rarely fully appreciate the forces that determine and shape policy implementation.

This schema is worth revisiting in the context of the typology of HIA, i.e. mandated, decision support, advocacy, and community empowerment, because it may provide an alternate lens for understanding why HIAs gain traction or not.