A HISTORY OF STUDENT RATINGS: Meso-Responserate Era (2000–2010)
The first decade of the new millennium rode on the search engines of the previous decade. A bunch of publications kicked it off with half a dozen edited volumes on student ratings and faculty evaluation published by Jossey-Bass in their Teaching and Learning series (Ryan, 2000, #83; Lewis, 2001, #87; Knapper & Cranton, 2002, #88; Sorenson & Johnson, 2004, #96) and Institutional Research series (Theall, Abrami,& Mets, 2001, #109; Colbeck, 2002, #114).
There were no planet-shattering technical developments, although there were several software options created specifically for online administration and reporting. Only a trickle (OOPS! Sorry. This is just residual fluid from the previous metaphor.) of articles on midterm formative ratings and other topics appeared. Finally, three books hit the Amazon pages in the last half of the decade: Arreola’s (2007)14th edition of his popular work, Seldin’s (2006) 128th edited volume, and our hero’s (Moi, 2006) psychometric-humorous attempt (see References in next blog).
Most of the activity and discourse on student ratings concentrated on practical issues. There were several trends that continued from the previous decade:
(1) student ratings data were being supplemented with other data, particularly peer review of teaching and course materials and letters of recommendation by each professor’s mommy or daddy, for decisions about teaching effectiveness;
(2) institutions reviewed the quality of their tools to consider either a commercial package, such as THOUGHT, or develop their own “homegrown” scales with online reporting by Academic Management Systems or other support;
(3) the technical quality of many “homegrown” scales prompted me to coin the new psychometric term "putrid"; and
(4) the debate over paper-based vs. online administration grew with cost and student response rates as the deal-breakers. Online packages were available everywhere. The importance of response rates to the validity of student ratings became so critical to the adoption of an online system that this era was named the "Meso-Responserate Era."
We’ve come to the end of this series. I’ve run out of jokes. The last blog will be an epilogue of a few final thoughts and jokes.
COPYRIGHT © 2010 Ronald A. Berk, LLC
Based on the Top 10 List of Strategies in my previous blog, which ones did you pick as MOST EFFECTIVE? Oh, you didn't read my previous blog. Shame, shame. Is this your 1st blog reading? Well, congrats! Welcome to Berk's BlogWorld. I hope you find it informative and fun!
For those of you who did peruse my Top 10, what did you pick? Let’s examine the evidence and see whether you get the prize. (SIDEBAR: “Yo, Blogboy, you didn’t say a prize was involved!” Oops, sorry. It was an afterthought.)
To date, the evidence indicates that students must believe (Do you believe? Kinda like Peter Pan!) that the results will be used for important decisions about faculty and courses in order for the online system to be successful. When faculty assign students in class to complete the evaluation with or without incentives, response rates are high. Finally, withholding early access to grades is supported by students as highly effective, but not too restrictive. This “withholding” approach has been very successful at raising response rates to the 90s at several institutions.
These strategies have boosted response rates into the 80s and 90s. It is a combination of elements, however, as suggested above, that must be executed properly to assure a high rate of students' return on the online investment. The research base and track record of online student ratings belies “low response rate” as an excuse for not implementing such a system in any institution.
While the combo strategy is recommended, there is one BEST technique among the top 10 that can really spike your response rate. It will be examined in the next blog.
Let me know if you have tried other strategies that have worked in your evaluation system.
COPYRIGHT © 2009 Ronald A. Berk, LLC
Here is a list of the Top 10 most effective strategies to get 80%+ response rates for online student ratings (Berk, 2006; Johnson, 2003; Sorenson & Reiner, 2003):
10. Faculty communicate online and face-face to students the importance of their input and how the results will be used
9. Ease of computer access for all students
8. Assurance of anonymity
7. Convenient, user-friendly online system
6. Provide easy-to-understand instructions on how to use the system
5. Administration and faculty strongly encourage students to complete forms
4. Faculty “assign” students to complete forms
3. Faculty threaten to smash students’ iPods with a Gallagher-type sledgehammer
3. System withholds students’ early access to final grades
2. Faculty provide extra credit or points
1. Faculty provide positive incentives (aka assorted bribes, such as requiring students to pick-up dry cleaning and take dog to the vet)
This is quite a laundry list of techniques. Is there any single practice that has a proven track record of success? Yup. That one is even more top-secret and a higher security “code chartreuse” technique than the others. It will be revealed in my next blog. Stay tuned.
COPYRIGHT © 2009 Ronald A. Berk, LLC
There have been cries, yelps, screams, shrieks, screeches, shrills, howls, and other sounds reverberating throughout the academic hallways and byways about low response rates from online administrations of student rating scales. Traditionally, the rates for in-class paper-based administrations following appropriate standardized procedures have been 80%+.
SAMPLING BIAS
First, let’s be Jack Nicholson A Few Good Men “crystal” clear about the problem with low response rates. When ratings dip below 80%, you’re in deeeep trouble! WAIT! That’s not the ending to that sentence. Where did it go? Oh, here it is: sampling bias increases, so that the summarized ratings will present an unrepresentative, biased picture of teaching performance in a given course. Such ratings could be inflated or deflated due to bias. This is evil, especially when student ratings constitute the only source of evidence on teaching performance, which is the case at many institutions. The results may be useless for either formative or summative decisions.
VALIDITY PROBLEM
The intractable problem is that there is no way to detect the direction or degree of bias. This low degree of rating score validity makes it extremely difficult to interpret the ratings based on the limited sample of students who chose (or self-selected themselves) to complete the scales. Keep in mind, these inaccurate ratings may still yield a high degree of internal consistency reliability, which can be misleading.
REASONS FOR NONRESPONSE
The response rate for online administrations can be half the rate or lower of paper-based administrations. This is a frequent objection to online ratings reported in faculty surveys. This fear of low response rate has deterred some institutions from adopting an online system. The research on this topic indicates the following possible reasons for the nonresponses: student apathy, perceived lack of anonymity, inconvenience, inaccessibility, technical problems, time for completion, and perceived lack of importance (Ballantyne, 2000; Dommeyer, Baum & Hanna, 2002; Sorenson & Reiner, 2003).
That’s the problem. Now how do we fix it? Several institutions have tested a variety of strategies to increase response rate that address several of the aforementioned reasons. These strategies have been suggested by faculty AND students. My next blog will present the Top 10 most effective strategies. Stick around as the plot thickens.
COPYRIGHT © 2009 Ronald A. Berk, LLC
