The letters in coloured circles problem: Student evaluation lets us improve, but only the things we can change

Five faces ranging from smiling to frowning, representing a scale

I found this post about student evaluations deeply affecting (CW: self-harm, general realness). The author’s experience resonated.

Every class, as we waited to start, I would ask my students (blue squares on the video conference screen) if they had any gossip. “I need something,” I said. “Any catfights? Anyone get really drunk and moon their neighbors? Anyone’s roommate cheat on their boyfriend? I am desperate here.”

Sometimes there would be a “no :(” in the chat.

Usually just silence.

Response to Student Evaluations, Sarah E. Smith

Online meetings and tutorials where nobody else turns on their screens have been the leitmotif of the past eighteen months. Online discussions that end up becoming discussions only between the teaching assistant and I, while some letters in coloured circles look on.

It’s not dehumanising exactly. It’s… nothing?

Why don’t they turn on their cameras? I don’t know. They might find it emotionally draining to meet via Zoom constantly. They might have a crappy internet connection. They might be doing other things. They might be self-conscious.

It’s not just students. Most meetings with staff are the same. A shared screen and a sea of letters in circles.

Connection seems to be missing. It’s not the medium, and it’s not solely the COVID pandemic. It’s something else.

Elaina Nguyen from Canada and fellow students from Australia, Hong Kong and Malaysia have argued that the values underpinning effective learning partnerships – respect, reciprocity, shared responsibility, authenticity, honesty, responsibility, inclusivity, reciprocity, empowerment, trust, courage, and plurality – all need to renegotiated not just for virtual environments, but also for our COVID world . They’re onto something.

Reading Sarah E. Smith’s post reminded me that some universities in Australia plan to publish subject evaluation metrics. There are proposals that students should be able to see aggregated subject evaluations, which are usually mean agreement scores with statements along the lines of “this subject enabled me to develop an understanding of the subject matter” or “this subject made me feel like part of a community”. Fair enough questions and useful information for teachers.

Student evaluation is unequivocally a good thing. It one of the main ways we can learn about what courses are like for students, and improve the way we teach.

The challenge is that often it’s not solely the teaching, the assessment activities or learning materials that students are evaluating.

Jenna Price captured the challenge at the heart of this in an article yesterday:

When students fill out those forms, they don’t answer the questions before them. La Trobe University lecturer Troy Heffernan has published widely on what students really mean when they rate subjects and the teachers who teach them. It’s about gender, race, accent. It’s also about institutional practices.

What goes wrong when uni students mark their teachers

Evaluation is only helpful if we’re able to act on the insights it offers. If students are using them to express dissatisfaction with student-to-teacher ratios, institutional policies, or have unacknowledged gender or racial biases , their usefulness evaporates. Similarly, evaluation scales become less useful for ongoing improvement if used as proxy metrics for poorly defined staff performance outcomes.

The way forward is clearly to develop more collaborative pedagogies based on students as partners , including co-creation of subjects and recognising students as consultants in that process.

At this time, what we need to improve as teachers probably isn’t captured through an agreement scale, valuable as that could be in the proper context. It’s solving the letters in coloured circles problem. It’s figuring out what’s really going on for the humans behind the circles.

I’d like to add one question to the student evaluation forms: why don’t they turn on their cameras?

References

Goodyear, P. (2005). Educational design and networked learning: Patterns, pattern languages and design practice. Australasian Journal of Educational Technology, 21(1). https://doi.org/10.14742/ajet.1344
Gleeson, J., Rickinson, M., Walsh, L., Cutler, B., Salisbury, M., Hall, G., & Khong, H. (2023). Quality use of research evidence: practitioner perspectives. Evidence & Policy, 1(aop), 1–21. https://doi.org/10.1332/174426421X16778434724277
Harris-Roxas, B., & Sturgiss, E. (2023). Equity in primary health care provision: more than 50 years of the inverse care law. Australian Journal of Primary Health, 29(2), i–ii. https://doi.org/10.1071/PY23062
Juarrero, A. (2002). Dynamics in action: Intentional behavior as a complex system (First MIT paperback edition). MIT.
George, B., Desmidt, S., Cools, E., & Prinzie, A. (2018). Cognitive styles, user acceptance and commitment to strategic plans in public organizations: an empirical analysis. Public Management Review, 20(3), 340–359. https://doi.org/10.1080/14719037.2017.1285112
Lee, D., McGuire, M., & Kim, J. H. (2018). Collaboration, strategic plans, and government performance: the case of efforts to reduce homelessness. Public Management Review, 20(3), 360–376. https://doi.org/10.1080/14719037.2017.1285113
Cepiku, D., Giordano, F., & Savignon, A. B. (2018). Does strategy rhyme with austerity? Public Management Review, 20(3), 421–443. https://doi.org/10.1080/14719037.2017.1285116
Spee, A. P., & Jarzabkowski, P. (2011). Strategic planning as communicative process. Organization Studies, 32(9), 1217–1245. https://doi.org/10.1177/0170840611411387
Klindt, M. P., Baadsgaard, K., & Jørgensen, H. (2023). Boundary spanning and partnership performance: bringing the structural perspective into the game. Public Management Review, 1–26. https://doi.org/10.1080/14719037.2023.2197914
Bryson, J. M., Edwards, L. H., & Van Slyke, D. M. (2018). Getting strategic about strategic planning research. Public Management Review, 20(3), 317–339. https://doi.org/10.1080/14719037.2017.1285111
assumptions of order, of rational choice, and of intent. (n.d.).
Proctor, E. K., Powell, B. J., Baumann, A. A., Hamilton, A. M., & Santens, R. L. (2012). Writing implementation research grant proposals: ten key ingredients. Implementation Science, 7(1), 96. https://doi.org/10.1186/1748-5908-7-96
Sax, M., Stinson, J., Stienstra, D., Levac, L., & Tatham, R. (2021). Environmental scan to identify domestic and international good practices to integrate SGBA+ in Health Impact Assessment. Canadian Research Institute for the Advancement of Women. https://www.criaw-icref.ca/wp-content/uploads/2021/05/SGBA_in_HIA_Final_Report.pdf
Nielsen, A. F., Michelmann, J., Akac, A., Palts, K., Zilles, A., Anagnostopoulou, A., & Langeland, O. (2023). Using the future wheel methodology to assess the impact of open science in the transport sector | Scientific Reports. Scientific Reports, 13(1), 6000. https://doi.org/10.1038/s41598-023-33102-5
Angeles, M. R., Crosland, P., & Hensher, M. (2023). Challenges for Medicare and universal health care in Australia since 2000. Medical Journal of Australia, 218(7). https://www.mja.com.au/journal/2023/218/7/challenges-medicare-and-universal-health-care-australia-2000?janus_id=b.harris-roxas@unsw.edu.au
Douglas, K. A., Dykgraaf, S. H., & Butler, D. C. (2023). Harnessing fast and slow thinking to ensure sustainability of general practice and functional universal health coverage in Australia. Medical Journal of Australia, Online first. https://www.mja.com.au/journal/2023/218/7/harnessing-fast-and-slow-thinking-ensure-sustainability-general-practice-and?janus_id=b.harris-roxas@unsw.edu.au
Callander, E. J. (2023). Out‐of‐pocket fees for health care in Australia: implications for equity. Medical Journal of Australia, Online first. https://www.mja.com.au/journal/2023/218/7/out-pocket-fees-health-care-australia-implications-equity?janus_id=b.harris-roxas@unsw.edu.au
Harris, E., & Harris, M. F. (2022). An exploration of the inverse care law and market forces in Australian primary health care. Australian Journal of Primary Health. https://doi.org/10.1071/PY22160
MRFF. (2023). Principles for Consumer Involvement in Research Funded by the Medical Research Future Fund - Advice from the Medical Research Future Fund Consumer Reference Panel. Medical Research Future Fund. https://www.health.gov.au/sites/default/files/2023-03/principles-for-consumer-involvement-in-research-funded-by-the-medical-research-future-fund.pdf
DPE. (2021, August 30). Understanding the effects of the 2019–20 fires. NSW Department of Planning and Environment. http://www.environment.nsw.gov.au/topics/fire/park-recovery-and-rehabilitation/recovering-from-2019-20-fires/understanding-the-impact-of-the-2019-20-fires
Snowden, D., & Boone, M. (2007). A Leader’s Framework for Decision Making. Harvard Business Review, 85(11), 68.
Thomas, P. A., Kern, D. E., & Hughes, M. T. (2022). Curriculum Development for Medical Education A Six-Step Approach (4th ed). Johns Hopkins University Press.
Sola, O., Kothari, P., Mason, H. R. C., Onumah, C. M., & Sánchez, J. P. (2019). The Crossroads of Health Policy and Academic Medicine: An Early Introduction to Health Policy Skills to Facilitate Change. MedEdPORTAL, 10827. https://doi.org/10.15766/mep_2374-8265.10827
Lucas, R., Kothari, P., Adams, C., Jones, L., Williams, V. N., & Sánchez, J. P. (2020). We are All Leaders: Introducing Self-Leadership Concepts Through the Lens of Improving Diversity in the Health Care Workforce. MedEdPORTAL, 11011. https://doi.org/10.15766/mep_2374-8265.11011
Falusi, O., Chun-Seeley, L., De La Torre, D., Dooley, D. G., Baiyewu, M., Gborkorquellie, T. T., Merrill, C. T., Davis, E., & Ward, M. C. (2023). Teaching the Teachers: Development and Evaluation of a Racial Health Equity Curriculum for Faculty. MedEdPORTAL, 11305. https://doi.org/10.15766/mep_2374-8265.11305
McEntee, A., Hines, S., Trigg, J., Fairweather, K., Guillaumier, A., Fischer, J., Bonevski, B., Smith, J. A., Wilson, C., & Bowden, J. (2022). Tobacco cessation and screening in CALD communities. The Sax Institute. https://doi.org/10.57022/sneg4189
Petticrew, M., Cummins, S., Ferrell, C., Findlay, A., Higgins, C., Hoy, C., Kearns, A., & Sparks, L. (2005). Natural experiments: an underused tool for public health? Public Health, 119(9), 751–757. https://doi.org/10.1016/j.puhe.2004.11.008
McDonald, F., Duckett, S., & Campbell, E. (2023). Commonwealth Power to Improve Access, Quality, and Efficiency of Medical Care: Does section 51 (xxiiiA) of the Constitution Limit Politically Feasible Health Policy Options Today? Federal Law Review, 0067205X231165872. https://doi.org/10.1177/0067205X231165872
Aromataris, E., & Munn, Z. (Eds.). (2020). JBI Manual for Evidence Synthesis. The Joanna Briggs Institute. https://reviewersmanual.joannabriggs.org/
Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., … Straus, S. E. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850
Ford, J., Vohra, S., Purdy, J., Walton, H., Ekermawi, R., O’Mullane, M., Birley, M., Douglas, M., Stevenson, A., Haigh, F., Harris-Roxas, B., Green, L., Black, D., Cave, B., Holroyd, I., & Lynch, H. (2023). Health Impact Assessments – current state of play and future direction (Grant FR-0004051) (Patent No. $44,750 (£24,000) (Cat 3)).
Brooks, F. P. (1995). The mythical man-month: essays on software engineering (Anniversary edition). Addison-Wesley Publishing Company. http://www.gbv.de/dms/bowker/toc/9780201835953.pdf
Bjerke, M. B., & Renger, R. (2017). Being smart about writing SMART objectives. Evaluation and Program Planning, 61, 125–127. https://doi.org/10.1016/j.evalprogplan.2016.12.009
Foroughi, Z., Ebrahimi, P., Aryankhesal, A., Maleki, M., & Yazdani, S. (2022). Toward a theory-led meta-framework for implementing health system resilience analysis studies: a systematic review and critical interpretive synthesis. BMC Public Health, 22(1), 287. https://doi.org/10.1186/s12889-022-12496-3
Rabin, B. A., & Brownson, R. C. (2017). Terminology for Dissemination and Implementation Research. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and Implementation Research in Health: Translating Science to Practice (pp. 19–46). Oxford University Press. https://doi.org/10.1093/oso/9780190683214.003.0002
Hailemariam, M., Bustos, T., Montgomery, B., Barajas, R., Evans, L. B., & Drahota, A. (2019). Evidence-based intervention sustainability strategies: a systematic review. Implementation Science, 14(1), 57. https://doi.org/10.1186/s13012-019-0910-6
Booth, A., Noyes, J., Flemming, K., Moore, G., Tunçalp, Ö., & Shakibazadeh, E. (2019). Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Global Health, 4(Suppl 1), e001107. https://doi.org/10.1136/bmjgh-2018-001107
Short, S. D., Marcus, K., & Balasubramanian, M. (2016). Health workforce migration in the Asia Pacific: Implications for the achievement of sustainable development goals. Asia Pacific Journal of Health Management, 11(3), 58–64. https://doi.org/10.3316/informit.536628016461757
Cairney, P., Boaz, A., & Oliver, K. (2023). Translating evidence into policy and practice: what do we know already, and what would further research look like? BMJ Quality & Safety. https://doi.org/10.1136/bmjqs-2023-015911
Peters, D. T., & Giacumo, L. A. (2020). Ethical and Responsible Cross-Cultural Interviewing: Theory to Practice Guidance for Human Performance and Workplace Learning Professionals. Performance Improvement, 59(1), 26–34. https://doi.org/10.1002/pfi.21906
RACGP. (2023). Position Statement: Artificial intelligence in primary care. Royal Australian College of General Practitioners. https://www.racgp.org.au/advocacy/position-statements/view-all-position-statements/clinical-and-practice-management/artificial-intelligence-in-primary-care
O’Mullane, M., Smith, K., Archibong, U., McHugh, S., Mullally, G., Purdy, J., Pursell, L., Harris-Roxas, B., Kelly, I., Kavanagh, P., Daly, H., O’Mahony, T., Green, L., Ward, J., Burke, S., Connolly, B., & Cave, B. (2023). Development of a Health Impact Assessment (HIA) Implementation Model: Enhancing Intersectoral Approaches in Tackling Health Inequalities (Patent No. $740,232 (€510,058) (Cat 3)).
WentWest. (2018). PHN Needs Assessment - Western Sydney Primary Health Network 2019-2022. WentWest Western Sydney Primary Health Network. https://wentwest.com.au/wp-content/uploads/planning_resources/Reports_WSPHN_Needs-Assessment_201921.pdf
WentWest. (2020). WentWest Western Sydney Primary Health Network Strategic Plan 2020-2023. WentWest Western Sydney Primary Health Network. https://wentwest.com.au/wp-content/uploads/Strategic_plan/Strategic-Plan_2020-2023_web.pdf
Ng, Z. Y. (2022). Family-centred care in early intervention audiology and speech-language pathology services for culturally and linguistically diverse (CALD) families of children with hearing loss [PhD, School of Health and Rehabilitation Sciences, University of Queensland]. https://doi.org/10.14264/715a67e
Kwok, R. (2013). Communication: Two minutes to impress. Nature, 494(7435), 138–138. https://doi.org/10.1038/nj7435-137a
Hekkert, M. P., Janssen, M. J., Wesseling, J. H., & Negro, S. O. (2020). Mission-oriented innovation systems. Environmental Innovation and Societal Transitions, 34, 76–79. https://doi.org/10.1016/j.eist.2019.11.011
Gibson, D., Willis, E., Merrick, E., Redley, B., & Bail, K. (2022). High demand, high commitment work: What residential aged care staff actually do minute by minute: A participatory action study. Nursing Inquiry, e12545. https://doi.org/10.1111/nin.12545
Pearson, R., Mullan, J., Dijkmans-Hadley, B., Halcomb, E., & Bonney, A. (2021). Medical care in Australian residential aged care: Perspectives of residents, family, nurses and general practitioners. Health & Social Care in the Community, 29(6), e279–e287. https://doi.org/10.1111/hsc.13352
Walton, V., Hogden, A., Long, J. C., Johnson, J. K., & Greenfield, D. (2019). How Do Interprofessional Healthcare Teams Perceive the Benefits and Challenges of Interdisciplinary Ward Rounds. Journal of Multidisciplinary Healthcare, 12, 1023–1032. https://doi.org/10.2147/JMDH.S226330