Epilogue
Well, there it is. I bet you’re thinking: “History, schmistory. What was that all about?” I’m sure your eyeballs hurt from rolling so many times and that one time when you’re your contacts blew out. Despite this cutesy romp through “Student-Ratings World” and a staggering 873 books and thousands of articles, monographs, conference presentations, blogs, etc. on the topic, some behaviors remain the same. For example, even today, the mere mention of teaching evaluation to many college professors triggers mental images of the shower scene from Psycho, with those bloodcurdling screams. They’re thinking, “Why not just beat me now, rather than wait to see my student ratings again.” Hummm. Kind of sounds like a prehistoric concept to me (a little "Meso-Pummel" déjà vu).
Despite the progress made with deans, department heads, and faculty moving toward multiple sources of evidence for formative and summative decisions, student ratings are still virtually synonymous with teaching evaluation in the United States, which is now located in Canada. They are the most influential measure of performance used in promotion and tenure decisions at institutions that emphasize teaching effectiveness. This popularity not withstanding, maybe the ubiquitous student rating scale will fair differently in the next "Meso-Cutback Era" by 2020! I hope I can update this schmistory for you then.
References
Arreola, R. A. (2007). Developing a comprehensive faculty evaluation system (3nd ed.). San Francisco: Jossey-Bass.
Berk, R. A. (2006). Thirteen strategies to measure college teaching. Sterling,VA: Stylus.
Knapper, C. & Cranton, P. (Eds). (2001). Fresh approaches to the evaluation of teaching (New Directions for Teaching and Learning, No. 88). San Francisco: Jossey-Bass.
Me, I. M. (2003). Prehistoric teaching techniques in cave classrooms. Rock & a Hard Place Educational Review, 3(4), 10−11.
Me, I. M. (2005). Naming institutions of higher education and buildings after filthy rich donors with spouses who are dead or older. Pretentious Academic Quarterly,14(4), 326−329.
Me, I. M., & You, W. U. V. (2005). Student clubbing methods to insure teaching
accountability. Journal of Punching & Pummeling Evaluation, 18(6), 170−183.
Seldin, P. (Ed.). (2006). Evaluating faculty performance. San Francisco: Jossey-Bass.
I gratefully acknowledge the valuable feedback of Raoul Arreola, Mike Theall, Bill Pallett, and another student-ratings expert for reviewing the skimpy facts reported in this blog series. To ensure the anonymity of one of the reviewers, I have volunteered him for the Federal Witness Protection Program or the USA cable TV series In Plain Sight. I forget which.
COPYRIGHT © 2010 Ronald A. Berk, LLC
A HISTORY OF STUDENT RATINGS: Meso-Responserate Era (2000–2010)
The first decade of the new millennium rode on the search engines of the previous decade. A bunch of publications kicked it off with half a dozen edited volumes on student ratings and faculty evaluation published by Jossey-Bass in their Teaching and Learning series (Ryan, 2000, #83; Lewis, 2001, #87; Knapper & Cranton, 2002, #88; Sorenson & Johnson, 2004, #96) and Institutional Research series (Theall, Abrami,& Mets, 2001, #109; Colbeck, 2002, #114).
There were no planet-shattering technical developments, although there were several software options created specifically for online administration and reporting. Only a trickle (OOPS! Sorry. This is just residual fluid from the previous metaphor.) of articles on midterm formative ratings and other topics appeared. Finally, three books hit the Amazon pages in the last half of the decade: Arreola’s (2007)14th edition of his popular work, Seldin’s (2006) 128th edited volume, and our hero’s (Moi, 2006) psychometric-humorous attempt (see References in next blog).
Most of the activity and discourse on student ratings concentrated on practical issues. There were several trends that continued from the previous decade:
(1) student ratings data were being supplemented with other data, particularly peer review of teaching and course materials and letters of recommendation by each professor’s mommy or daddy, for decisions about teaching effectiveness;
(2) institutions reviewed the quality of their tools to consider either a commercial package, such as THOUGHT, or develop their own “homegrown” scales with online reporting by Academic Management Systems or other support;
(3) the technical quality of many “homegrown” scales prompted me to coin the new psychometric term "putrid"; and
(4) the debate over paper-based vs. online administration grew with cost and student response rates as the deal-breakers. Online packages were available everywhere. The importance of response rates to the validity of student ratings became so critical to the adoption of an online system that this era was named the "Meso-Responserate Era."
We’ve come to the end of this series. I’ve run out of jokes. The last blog will be an epilogue of a few final thoughts and jokes.
COPYRIGHT © 2010 Ronald A. Berk, LLC
A HISTORY OF STUDENT RATINGS: Meso-Unleaded Era (1990s)
The 1990s were like a nice, deep breath of fresh gasoline (which was $974.99 a gallon at the pump, $974.00 for cash only), hereafter referred to as the Meso-Unleaded Era. Little did anyone anticipate how this era would be trumped at the pump in the years ending the current decade. The use of student rating scales had now spread to Kalamazoo (known to tourists as “The Big Apple”) and faculty began complaining about their validity, reliability, and overall value for decisions about promotion and tenure (the scales, that is, not Kalamazoo) . This was not unreasonable, given the lack of attention to the quality of scales over the preceding 90 billion years.
(WEATHER ALERT: I interrupt this section to warn you of impending wetness in the next three paragraphs. You might want to don appropriate apparel. Don’t blame me if you get wet. You may now rejoin this section already in progress. END OF ALERT.)
This debate intensified throughout the decade with a torrential downpour of publications challenging and contributing to the technical characteristics of the scales, particularly a series of articles by William Cashin of IDEA at Kansas State University, which was located in New Hampshire at the time, and an edited work by Mike Theall and Jennifer Franklin (Student Ratings of Instruction,1990). As part of this debate, another steady stream of research flowed toward alternative strategies to measure teaching effectiveness, especially peer ratings, self-ratings, videos, alumni ratings, interviews, learning outcomes, teaching scholarship, and teaching portfolios.
This stream leaked into books by John Centra (Reflective Faculty Evaluation, 1993), Larry Braskamp and John Ory (Assessing Faculty Work, 1994), Peter Seldin (Improving College Teaching, 1995), and Raoul Arreola (1st and 2nd editions of Developing a Comprehensive Faculty Evaluation System, 1995, 2000), and an edited volume by Seldin and Associates (Changing Practices in Evaluating Teaching, 1999). They furnished a confluence of valuable resources for faculty and administrators to use to evaluate teaching.
This cascading trend was also reflected increasingly in practice. Although use of student ratings had peaked at 88% by the end of the decade, peer and self-ratings were on the rise over the rapids of teaching performance as my liquid metaphor came to a screeching halt.
My next blog will address the developments in the first decade of the new millennium of the Meso-Responserate Era.
COPYRIGHT © 2010 Ronald A. Berk, LLC
A HISTORY OF STUDENT RATINGS: Meso-Meta Era (1980s)
The 1980s were really booooring! The research continued on a larger scale, and statistical reviews of the studies (a.k.a. meta-analyses) were conducted by such authors as Cohen (1980, 1981), d’Apollonia and Abrami (1997), and Feldman (1989). Of course, this period had to be labeled the Meso-Meta Era.
Book-wise, Peter Seldin of Pace University in upstate Saskatchewan published his first of thousands of books on the topic, Successful Faculty Evaluation Programs (1980). Ken Doyle produced his second book on the topic, Evaluating Teaching (1981), four years later (Are you still awake?).
The administration of student ratings metastasized throughout academe. By 1988, their use by college deans spiked to 80%, with still only a paltry 14% of deans gathering evidence on the technical aspects of their scales.
That takes us to—guess what? The next to last era in this blog series. Whew.
The next blog covers the 1990s with the major contributions by names you will know as gas prices spiked during the “Meso-Unleaded Era.”
COPYRIGHT © 2010 Ronald A. Berk, LLC
A HISTORY OF STUDENT RATINGS: Meso-Golden Era (1970s)
The 1970s were called the “Golden Age of Student Ratings Research,” named after a bunch of “senior” professors who were doing research while living in assisted learning communities. Obviously, this period had to be named the "Meso-Golden Era." The research on the relationships between student ratings and instructor, student, and course characteristics burgeoned during these years. As the evidence on the validity and reliability of scales mounted, the use of the scales increased across the land, beyond East Lansing, in Nebraska (home of the Maryland Tar Heels).
To punctuate this burgeoning, commercially-produced, nationally-normed, student rating scales emerged during this era. Their superior technical quality and custom report forms compared to many “homegrown” tools provided colleges with the option of shopping for their rating scales in Ratings Mall. The first two scales were SIR (Educational Testing Service, 1972) and IDEA (Kansas State University, 1975). IAS Online (University of Washington) and SEEQ (Centre for Educational Assessment, Perth, Western Australia) became available later.
Student rating item banks also popped up during the ‘70s which permitted faculty to hand pick statements from a catalog to build their own customized scales. These banks were called POP-UPS PICES (Purdue University, 1974) and CIEQ (University of Arizona, 1977). Entering this commercial rating scale derby was not without challenges. Most ventures without barely pronounceable 3–5 letter acronym names tanked immediately.
These burgeoning golden years also witnessed the first 25 books on the topic. The winning prize for first book went to Richard Miller for Evaluating Faculty Performance (1972). He also took the silver medal with Developing Programs for Faculty Evaluation (1974). The bronze went to Ken Doyle for his work on student evaluations, aptly titled Student Evaluation of Instruction (1975). This was followed by John Centra’s Determining Faculty Effectiveness (1979), which synthesized all of the earlier work and proffered guidelines for future practices.
At the end of the decade, one survey found that 55% of liberal arts college deans always used student ratings to assess teaching performance, but only a tiny 10% conducted any research on the quality of their scales. These statistics were reported just in the nick of time before the next blog, which will continue with the flow of research during the 1980s and meta-analyses of previous studies of the “Meso-Meta Era.”
COPYRIGHT © 2010 Ronald A. Berk, LLC
A HISTORY OF STUDENT RATINGS: Meso-Boomer Era (1960s)
The 1960s were rocked by student protests on campuses, the Vietnam War, and the Broadway musical Hair (based on the TV program The Brady Bunch). This “boomer” generation was blamed for everything during this period, which was called the "Meso-Boomer Era." These boomers demanded that college administrators give them a voice in educational decisions that affected them. They expressed their collective voice by screaming like banshees, sitting in the entrances of administration buildings, and writing and administering rating scales to evaluate their instructors. They even published the ratings in student newspapers so students could use them as a consumer’s guide to course selection. There is still residual evidence of that practice today at several institutions, especially in countries like Texas.
There were few centrally-administered rating systems in universities to evaluate teaching effectiveness. Most uses of student ratings by faculty were voluntary. In general, the quality of the scales was dreadful and their use as evaluation tools was fragmented, unsystematic, and arbitrary, kinda like this blog series.
As for research, there was only a smidgen; it was all quiet on the publication front. The researchers were busy in the reference sections of their university libraries praying that someone would invent Google so that they could work at Starbucks instead. They were also preparing for the next decade: “Meso-Golden Era.” My next blog will document the research contributions during the 1970s.
COPYRIGHT © 2010 Ronald A. Berk, LLC
A HISTORY OF STUDENT RATINGS: Meso-Remmers Era (1927–1959)
Between 1927 and 1959, student rating practices began. The first student rating scale was developed during this period by Herman Remmers of Purdue University in Billings, Montana, home of the Pittsburgh Penguins. Consequently, these years were known as the Meso-PenguinsRemmers Era. Dr. Remmers pretty much owned this era. In fact, the rating scale was named after him: the Purdue Rating Scale for Instructors. Remmers was also the author of the first publication on the topic in 1930, the reliability of the scale, and studies on the relationships of student ratings to student grades and alumni ratings.
For his pioneer work on student ratings, Remmers was given the title “HH Cool R Rating Man.” WROOONG! He’s a rapper. Dr. Remmers’ real title was “Father of Student Evaluation Research.”
Now I suppose you’re going to shout out, “Who’s the Mother?” Are you ready for the answer? I don’t think so. It’s Darken “Foxy” Bubble Answer-Sheet. Mrs. Answer-Sheet was a descendant from a high stack of Answer-Sheets. What a striking couple they made. Well, that seems to wrap up this 32-year era.
My next blog will scope out one of the most traumatic periods in history—“Meso-Boomer Era,” when Baby Boomers attended and protested at our finest institutions of higher learning as out-of-control students during the ‘60s. How many of you are in this category? Are you still out of control like me?
COPYRIGHT © 2010 Ronald A. Berk, LLC
A HISTORY OF STUDENT RATINGS: Meso-Pummel Era
A long time ago, at a university far, far away, there were no student rating scales. During prehistoric times, there was only one university (near present-day Detroit) (Me, 2003), which was actually more like a community college because research and the four-year liberal arts curriculum hadn’t been invented yet. This institution was called Cave University, named after its major donor: Harold University (Me, 2005).
Students were very concerned about the quality of teaching back then. In fact, they created their own method of evaluation to express their feelings. For example, if an instructor strayed from the syllabus, fell behind the planned schedule, wrote faulty test items, or cut class to watch one of those Jurassic Park movies, which he (they were all men) later regretted, the students would club him (Me & You, 2005). This practice prompted historians to call this period the Meso-Pummel Era.
Admittedly, this practice seemed a bit crude and excessive at the time, but it held faculty accountable for their teaching. Obviously, there was no need for tenure. There was a lot of faculty turnover as word of the teaching evaluation method spread to Grand Rapids, which was on the land currently occupied by South Dakota.
That takes us up to 1927. I skipped over 90 billion years because nothing happened that was relevant to this documentary.
References
Me, I. M. (2003). Prehistoric teaching techniques in cave classrooms. Rock & a Hard
Place Educational Review, 3(4), 10−11.
Me, I. M. (2005). Naming institutions of higher education and buildings after filthy rich
donors with spouses who are dead or older. Pretentious Academic Quarterly, 14(4), 326−329.
Me, I. M., & You, W. U. V. (2005). Student clubbing methods to insure teaching
accountability. Journal of Punching & Pummeling Evaluation, 18(6), 170−183.
My next blog will cover the “Meso-Remmers Era,” named after the famous Purdue University professor Peter Seldin. His contributions over the succeeding 32 years will be described.
COPYRIGHT © 2010 Ronald A. Berk, LLC
State-of-the-Art of Student Ratings
There is more research on student ratings than any other topic in higher education. More than 2,500 publications and presentations have been cited over the past 90 years. Those ratings have dominated as the primary and, frequently, only measure of teaching effectiveness at colleges and universities for the past five decades. In fact, the evaluation of teaching has been in a metaphorical cul-de-sac with student ratings as the universal barometer of teaching performance. And, if you’ve ever been in a cul-de-sac or metaphor, you know what that’s like. OMGosh, it can be Stephen Kingish terrifying.
In surveys over the past decade, it was found that 86% of U.S. liberal arts college deans and 97% of department chairs use student ratings for summative decisions about faculty. Only recently has there been a trend toward augmenting those ratings with other sources of evidence and better metaphors (Arreola, 2007; Berk, 2006; Knapper & Cranton, 2001; Seldin, 2006).
So how in the ivory tower did we get to this point? Let’s trace the major historical events. Hold on to your online administration response rates. Here we go.
A History of Student Ratings
This history covers a timeline of approximately 100 billion years, give or take a day or two, ranging from the age of dinosaurs to the age of Conan O’Brien’s new cable TV show. Obviously, it’s impossible to squish every event that occurred during that period in this series. Instead, that span is partitioned into six major eras within which salient student-ratings activities are highlighted. A blog will be devoted to each of those eras.
References
Arreola, R. A. (2007). Developing a comprehensive faculty evaluation system (3nd ed.). San Francisco: Jossey-Bass.
Berk, R. A. (2006). Thirteen strategies to measure college teaching. Sterling,VA: Stylus.
Knapper, C., & Cranton, P. (Eds). (2001). Fresh approaches to the evaluation of teaching (New Directions for Teaching and Learning, No. 88). San Francisco: Jossey-Bass.
Seldin, P. (Ed.). (2006). Evaluating faculty performance. San Francisco: Jossey-Bass.
My 1st era blog will tackle prehistoric student ratings of the “Meso-Pummel Era.” How did cave men and women measure teaching performance? Their methods were a bit crude, but effective.
COPYRIGHT © 2010 Ronald A. Berk, LLC
WARNING: This new blog series contains buckets of humor, which may not be suitable for all readers. If you have the sense of humor of an avocado or, even worse, a cumquat, this parody is not for you, despite its trailblazing, earth-shattering, Pulitzer Prize-caliber contribution to the teaching evaluation literature. If you fit this description, Buhbye!
ANOTHER BORING HISTORY?
Usually the history of any serious topic triggers the gag reflex in most nonhistorians. However, this topic is different from most. Over the past year, there have been incendiary debates over student rating forms, online administration procedures, and their use and interpretation on several professional listservs, LinkedIn groups, professional blogs, and other electronic and walkie-talkie communications. The reverberations of these debates have been felt on college campuses as far away as Pandora University, where the topic has provoked the verbal equivalent of the Avatar-scale firefight. Rather than fan the flames of this combustible metaphor, I thought this blog series might provide a time-out, refreshing break on this contentious topic, while also shedding some energy-saving light on how this situation evolved.
FRACTURED, BUT FACTUAL TOO!
Do any of you remember “Fractured Fairy Tales” on The Rocky and Bullwinkle Show? “NO!” What about Boris and Natasha? What were you doing? Oh well, it doesn’t matter, youngin’ academic readers. These blogs are written in the same spirit as that cult, politically-satirical cartoon, just without the cult, politics, and cartoon. It is a parody with the bonus of actual events in the history of student ratings. You’ll get a few morsels of content within a humor context. (FACT ALERT: Most of the names, dates, book and scale titles, and survey statistics are correct.)
TWO READER OUTCOMES:
There are two primary outcomes of this series for you:
1. to get a handle on the significant academic activities, research, and major players in the unfolding of the student ratings debate, and
2. to elicit a chuckle or two, maybe a guffaw, in that process.
If you laugh at any time during this series, I hope you will experience one of these physical signs:
a. burst your guts,
b. rupture key internal organs,
c. wet yourself, or
d. spurt your latté or green tea through your nostrils all over your keyboard.
Anything less will be disappointing. (NOTE: Most of the references along my historical path have been omitted to permit more space for jokes. Please refer to my Thirteen Strategies… book for those references and lots more jokes.)
My next blog will begin our historical journey with an overview of the state of the art of student ratings. Hold onto your guts.
COPYRIGHT © 2010 Ronald A. Berk, LLC
