Why Should GLOBAL ITEM SCORES Not Be Used for Summative Decisions? PART III

As a continuation of the last blog on the first two reasons for not using global items for summative decisions about faculty, this blog describes the third and most important reason:



3. PROFESSIONAL AND LEGAL STANDARDS: One or 2 global rating item scores alone for major summative decisions about faculty performance are totally inadequate. That administrative practice violates national testing/scaling practices according to the Standards for Educational and Psychological Testing and EEOC Uniform Guidelines on Employee Selection Procedures, plus the rulings from a large corpus of court cases on this topic. Essentially, it's ILLEGAL to make such personnel decisions about faculty. Clearly, these are PERSONNEL decisions about us, not instructional or curriculum decisions. In the case of employee decisions like these, 1 or 2 items do not reflect an accurate assessment of the instructor's job behaviors. A total scale score based on, for example, 35 items defining effective teaching behaviors, or subscale scores on specific areas of teaching competency would satisfy those criteria. A long history of court cases on personnel decisions indicates that the instrument used for personnel decisions must be based on a comprehensive job analysis of the job’s tasks related to a person’s knowledge, skills, and abilities (KSAs). The behaviors listed as items on the total scale satisfy that standard for teaching effectiveness.

Although administrators have used global items in some form for decisions about faculty teaching performance for quite some time, those practices should stop. They have lawsuit written all over them. As noted above, important, possibly career-changing, individual personnel decisions are held to the highest standards professionally and legally, as they should be. If the instructor being violated is a minority or female, be prepared for an EEOC offensive. If you know an administrator who is engaging in such practices, the recommendation is “cease and desist.”



What’s the alternative? Use the total scale score or subscale scores for different areas of performance in conjunction with other measures, such as peer evaluations, self-ratings, and a dozen other possible sources of evidence (for further details, see my October 25, 2009 blog and Thirteen Strategies… Stylus Publisher link in right margin).

Please let me know your thoughts and observations on my recommendations in these blogs.


COPYRIGHT © 2009 Ronald A. Berk, LLC

When Should Student Rating Scales Be Administered? Who Cares?

One critical psychometric issue neglected in the literature on administration of student rating scales and in wide-spread administration practices is the permissible window for completing the scales. This involves standardization of administration procedures. These procedures are required in the administration of all instruments, especially those used for personnel decisions in the Standards for Educational and Psychological Testing. Employment decisions about merit pay, pay cuts, promotion, demotion, and tenure must be based on evidence of teaching performance that meets the Standards and EEOC Uniform Guidelines on Employee Selection Procedures.

The validity of the responses, their comparability, and aggregate meaning based on group data hinge on WHEN the students complete the evals. The window must be NARROW, such as 48 hours immediately following the final exam or project submission. If there are items on the scale measuring student assessment procedures, administration practices, fairness, etc., the final exam must be completed before the scales can be administered. This STANDARDIZATION of scale administration is essential, whether online or paper-and-pencil, to insure the scores from all students have the same meaning. If a wide window is given where some students can complete rating scales before the final assessment and others after that assessment at their discretion, the validity of responses is shot to smithereens!

The issue is the link between the behaviors measured on the rating scale and the students' opportunity to render an accurate rating of each behavior? This is a validity concern. It is assumed that every student has had the same 45 hours (3-credit course) to observe those behaviors during the semester. If they miss a few classes, their evaluations should not be significantly affected.


COPYRIGHT © 2009 Ronald A. Berk, LLC