A HISTORY OF STUDENT RATINGS: Meso-Unleaded Era (1990s)
The 1990s were like a nice, deep breath of fresh gasoline (which was $974.99 a gallon at the pump, $974.00 for cash only), hereafter referred to as the Meso-Unleaded Era. Little did anyone anticipate how this era would be trumped at the pump in the years ending the current decade. The use of student rating scales had now spread to Kalamazoo (known to tourists as “The Big Apple”) and faculty began complaining about their validity, reliability, and overall value for decisions about promotion and tenure (the scales, that is, not Kalamazoo) . This was not unreasonable, given the lack of attention to the quality of scales over the preceding 90 billion years.
(WEATHER ALERT: I interrupt this section to warn you of impending wetness in the next three paragraphs. You might want to don appropriate apparel. Don’t blame me if you get wet. You may now rejoin this section already in progress. END OF ALERT.)
This debate intensified throughout the decade with a torrential downpour of publications challenging and contributing to the technical characteristics of the scales, particularly a series of articles by William Cashin of IDEA at Kansas State University, which was located in New Hampshire at the time, and an edited work by Mike Theall and Jennifer Franklin (Student Ratings of Instruction,1990). As part of this debate, another steady stream of research flowed toward alternative strategies to measure teaching effectiveness, especially peer ratings, self-ratings, videos, alumni ratings, interviews, learning outcomes, teaching scholarship, and teaching portfolios.
This stream leaked into books by John Centra (Reflective Faculty Evaluation, 1993), Larry Braskamp and John Ory (Assessing Faculty Work, 1994), Peter Seldin (Improving College Teaching, 1995), and Raoul Arreola (1st and 2nd editions of Developing a Comprehensive Faculty Evaluation System, 1995, 2000), and an edited volume by Seldin and Associates (Changing Practices in Evaluating Teaching, 1999). They furnished a confluence of valuable resources for faculty and administrators to use to evaluate teaching.
This cascading trend was also reflected increasingly in practice. Although use of student ratings had peaked at 88% by the end of the decade, peer and self-ratings were on the rise over the rapids of teaching performance as my liquid metaphor came to a screeching halt.
My next blog will address the developments in the first decade of the new millennium of the Meso-Responserate Era.
COPYRIGHT © 2010 Ronald A. Berk, LLC
A HISTORY OF STUDENT RATINGS: Meso-Golden Era (1970s)
The 1970s were called the “Golden Age of Student Ratings Research,” named after a bunch of “senior” professors who were doing research while living in assisted learning communities. Obviously, this period had to be named the "Meso-Golden Era." The research on the relationships between student ratings and instructor, student, and course characteristics burgeoned during these years. As the evidence on the validity and reliability of scales mounted, the use of the scales increased across the land, beyond East Lansing, in Nebraska (home of the Maryland Tar Heels).
To punctuate this burgeoning, commercially-produced, nationally-normed, student rating scales emerged during this era. Their superior technical quality and custom report forms compared to many “homegrown” tools provided colleges with the option of shopping for their rating scales in Ratings Mall. The first two scales were SIR (Educational Testing Service, 1972) and IDEA (Kansas State University, 1975). IAS Online (University of Washington) and SEEQ (Centre for Educational Assessment, Perth, Western Australia) became available later.
Student rating item banks also popped up during the ‘70s which permitted faculty to hand pick statements from a catalog to build their own customized scales. These banks were called POP-UPS PICES (Purdue University, 1974) and CIEQ (University of Arizona, 1977). Entering this commercial rating scale derby was not without challenges. Most ventures without barely pronounceable 3–5 letter acronym names tanked immediately.
These burgeoning golden years also witnessed the first 25 books on the topic. The winning prize for first book went to Richard Miller for Evaluating Faculty Performance (1972). He also took the silver medal with Developing Programs for Faculty Evaluation (1974). The bronze went to Ken Doyle for his work on student evaluations, aptly titled Student Evaluation of Instruction (1975). This was followed by John Centra’s Determining Faculty Effectiveness (1979), which synthesized all of the earlier work and proffered guidelines for future practices.
At the end of the decade, one survey found that 55% of liberal arts college deans always used student ratings to assess teaching performance, but only a tiny 10% conducted any research on the quality of their scales. These statistics were reported just in the nick of time before the next blog, which will continue with the flow of research during the 1980s and meta-analyses of previous studies of the “Meso-Meta Era.”
COPYRIGHT © 2010 Ronald A. Berk, LLC
CRITERION-REFERENCED SCORE INTERPRETATIONS
All of the preceding score interpretations compare a score rating to the score range and points along the scale. For example, any rating can be above or below the midpoint or even higher or lower on the respective scale in terms of degree of favorableness, such as illustrated in the previous blog. Cut-off scores can also be set for criterion-referenced interpretations.
NORM-REFERENCED SCORE INTERPRETATIONS
Alternatively or in addition to these score interpretations, ratings can be compared to scores by a norm group, such as those by instructors of similar courses and/or instructors in your department, school, or university. They can also be compared to regional or national norms of instructors who teach the same courses you teach. Commercially-produced scales, such as IDEA, offer those normative scores. These comparisons are called norm-referenced interpretations. The scores at the various levels are still the same; they’re just summarized and reported for different groups of instructors and courses.
STRUCTURED VS. UNSTRUCTURED RESULTS
Did you do really well? If not, you should be able to figure out why you didn't. The answers to the unstructured or open-ended questions at the end of your scale usually provide reasons to explain the responses to the structured student ratings. The structured and unstructured items yield complementary evidence of your performance. Now you probably have more information than you want. That’s why I’m here for you.
I hope the preceding 2 weeks of blogs on interpreting student rating results have helped a little to make sense out of the scores you received. At least, you may have a basic understanding of the possible scores that can be reported and how you can use them to improve your teaching.
Let me know if you have any questions about the material presented. Much more detail on these topics is covered in my Thirteen Strategies book.
HAPPY STUDENT RATINGS!!
COPYRIGHT © 2010 Ronald A. Berk, LLC
APPLICATION TO DIFFERENT FORMS
Although each of you is using a different rating form with different numbers of items and scores, those differences do not matter in score interpretation. Whether you’re using a commercial package, such as IDEA, SIR II, PICES, or CIEQ DU SOLEIL, or a “homegrown scale,” there are only so many score reporting possibilities for any form in Likert-type format. So my suggestions are generic and should be applicable to your form. I encourage you to consult the guidelines or manual for your reporting system for more specific information.
FIVE BASIC CATEGORIES OF RESULTS
There are 5 possible categories of results reported for most student rating forms:
1. anchor distribution of percentages
2. item statistics (mean and/or median)
3. subscale statistics (mean and/or median)
4. total scale statistics (mean and/or median)
5. summary of comments to open-ended questions
Your report form may not provide all of the above, but it should certainly give you at least 2 and 4.
WHAT DO FACULTY NEED?
That's a lot of information. You could use all of those results, however, 1 and 2, in particular, provide the most valuable diagnostic info to revise teaching or course materials that will benefit your next course-load of students. These are called formative decisions about teaching. Category 5 can explain the reasons for the ratings to 1 and 2.
WHAT DO ADMINISTRATORS NEED?
Summative decisions about annual contract renewal, merit pay, or promotion and tenure review by department chairs, associate deans, etc. can be based on 3 and 4 and possibly the global item scores.
This blog series will focus primarily on the faculty needs. My next blog will examine the 1st level of interpretation: ANCHOR-WORLD!
COPYRIGHT © 2010 Ronald A. Berk, LLC
