Should GLOBAL ITEM SCORES from Student Evaluations Be Used for Important Faculty Decisions? PART I

I know what you’re thinking: “I thought you were done with this student rating stuff. Get off it already.” I know, but I rethought my thought after receiving my weekly report that more people read my blogs this past week than at any other time over the past 3 months. Maybe I hit a note; maybe not. Anyway, these topics keep popping up on listservs and workshops.


But that’s not REEEAALLY what you were thinking. It is: “What in the world is a global item?” For those of you who are not of this world and are just passing through, as I am, here is a profile of the global item:

1. It provides a general, broad-stroke indication of teaching performance. It's intended to be an omnibus item, representing the collective judgments on all other items. But it isn't. Only the total scale score does that.
2. It doesn’t address specific teaching and course characteristics.
3. It usually appears at the end of the rating scale and should not be summed with the scores of all other items.

Using a "Strongly Agree-Strongly Disagree" anchor response scale, a couple of examples are given below:


Overall, my instructor is a dirtbag.
Overall, I learned squat in this course.


Of course, you know I’m kidding. Better items are:


Overall, my instructor is a moron.
Overall, this course is putrid.


Usually, there are 1 to 3 items. Frequently, administrators, such as your department chair, associate dean, or emperor or empress, will be encouraged to use the ratings on those items to provide a simple, quick-and-dirty measure of your teaching performance. Those ratings, in lieu of the total scale or subscale scores, are used in conjunction with other information to arrive at summative decisions regarding merit pay, contract renewal for full-time and adjunct faculty, and promotion and tenure recommendations. Are these important decisions about your career and life. You bet!!


Do you want those decisions to be rendered on the basis of 1 or 2 items? “Sure, why not?” Are you kidding me? There are several logical, psychometric, and legal reasons why global items should NOT be used for summative decisions. They will be described in my next blog. Stay tuned for more fun from RatingWorld.


COPYRIGHT © 2009 Ronald A. Berk, LLC

When Should Student Rating Scales Be Administered? Who Cares?

One critical psychometric issue neglected in the literature on administration of student rating scales and in wide-spread administration practices is the permissible window for completing the scales. This involves standardization of administration procedures. These procedures are required in the administration of all instruments, especially those used for personnel decisions in the Standards for Educational and Psychological Testing. Employment decisions about merit pay, pay cuts, promotion, demotion, and tenure must be based on evidence of teaching performance that meets the Standards and EEOC Uniform Guidelines on Employee Selection Procedures.

The validity of the responses, their comparability, and aggregate meaning based on group data hinge on WHEN the students complete the evals. The window must be NARROW, such as 48 hours immediately following the final exam or project submission. If there are items on the scale measuring student assessment procedures, administration practices, fairness, etc., the final exam must be completed before the scales can be administered. This STANDARDIZATION of scale administration is essential, whether online or paper-and-pencil, to insure the scores from all students have the same meaning. If a wide window is given where some students can complete rating scales before the final assessment and others after that assessment at their discretion, the validity of responses is shot to smithereens!

The issue is the link between the behaviors measured on the rating scale and the students' opportunity to render an accurate rating of each behavior? This is a validity concern. It is assumed that every student has had the same 45 hours (3-credit course) to observe those behaviors during the semester. If they miss a few classes, their evaluations should not be significantly affected.


COPYRIGHT © 2009 Ronald A. Berk, LLC