Showing posts with label
student evaluation of faculty.
Show all posts
Showing posts with label
student evaluation of faculty.
Show all posts
As a continuation of the last blog on the first two reasons for not using global items for summative decisions about faculty, this blog describes the third and most important reason:
3. PROFESSIONAL AND LEGAL STANDARDS: One or 2 global rating item scores alone for major summative decisions about faculty performance are totally inadequate. That administrative practice violates national testing/scaling practices according to the Standards for Educational and Psychological Testing and EEOC Uniform Guidelines on Employee Selection Procedures, plus the rulings from a large corpus of court cases on this topic. Essentially, it's ILLEGAL to make such personnel decisions about faculty. Clearly, these are PERSONNEL decisions about us, not instructional or curriculum decisions. In the case of employee decisions like these, 1 or 2 items do not reflect an accurate assessment of the instructor's job behaviors. A total scale score based on, for example, 35 items defining effective teaching behaviors, or subscale scores on specific areas of teaching competency would satisfy those criteria. A long history of court cases on personnel decisions indicates that the instrument used for personnel decisions must be based on a comprehensive job analysis of the job’s tasks related to a person’s knowledge, skills, and abilities (KSAs). The behaviors listed as items on the total scale satisfy that standard for teaching effectiveness.
Although administrators have used global items in some form for decisions about faculty teaching performance for quite some time, those practices should stop. They have lawsuit written all over them. As noted above, important, possibly career-changing, individual personnel decisions are held to the highest standards professionally and legally, as they should be. If the instructor being violated is a minority or female, be prepared for an EEOC offensive. If you know an administrator who is engaging in such practices, the recommendation is “cease and desist.”
What’s the alternative? Use the total scale score or subscale scores for different areas of performance in conjunction with other measures, such as peer evaluations, self-ratings, and a dozen other possible sources of evidence (for further details, see my October 25, 2009 blog and Thirteen Strategies… Stylus Publisher link in right margin).
Please let me know your thoughts and observations on my recommendations in these blogs.
COPYRIGHT © 2009 Ronald A. Berk, LLC
Please consider the following 2 reasons why the global items may not be the best option for those summative decisions:
1. ACCURACY AND FAIRNESS: After your students have spent 45 hours in your course over the semester, does their rating of 1 item seem to accurately capture the sum total of all of the experiences in your classroom? There is no doubt that the item furnishes information about your performance and the course, but should it be used for summative, super-important decisions about your career? Is 1 item score of 0-4 a fair and reasonable base to infer overall performance? Could you accurately and fairly rate the performance of your administrative assistant, department chair, or dean with 1 item to truly evaluate his or her degree of effectiveness?
2. VALIDITY AND RELIABILITY: Think of student rating scales like the S.A.T. One item from the S.A.T. doesn't provide a valid and reliable score of verbal ability any more than 1 or 2 global items on a rating scale provide a valid and reliable measure of teaching performance. Although the former measures knowledge and the latter measures attitudes or opinions, the psychometric problem is virtually identical. Such data would be tantamount to giving high schoolers 1 or 2 verbal and quantitative items from the S.A.T. and making individual college admission decisions on those scores. Those scores are technically unreliable. One or 2 items are extremely unreliable compared to a summary score derived from 5 subscales or 35 items. Reliability coefficients should be in the .80s-.90s for individual decisions, although coefficients as low as the .60s are acceptable for group-based decisions in research. Typically, a single item does not yield a coefficient in the acceptable range.
The third reason, PROFESSIONAL AND LEGAL STANDARDS, will be covered in the next blog. It is the most important of the 3 reasons. Don’t miss it.
COPYRIGHT © 2009 Ronald A. Berk, LLC
I know what you’re thinking: “I thought you were done with this student rating stuff. Get off it already.” I know, but I rethought my thought after receiving my weekly report that more people read my blogs this past week than at any other time over the past 3 months. Maybe I hit a note; maybe not. Anyway, these topics keep popping up on listservs and workshops.
But that’s not REEEAALLY what you were thinking. It is: “What in the world is a global item?” For those of you who are not of this world and are just passing through, as I am, here is a profile of the global item:
1. It provides a general, broad-stroke indication of teaching performance. It's intended to be an omnibus item, representing the collective judgments on all other items. But it isn't. Only the total scale score does that.
2. It doesn’t address specific teaching and course characteristics.
3. It usually appears at the end of the rating scale and should not be summed with the scores of all other items.
Using a "Strongly Agree-Strongly Disagree" anchor response scale, a couple of examples are given below:
Overall, my instructor is a dirtbag.
Overall, I learned squat in this course.
Of course, you know I’m kidding. Better items are:
Overall, my instructor is a moron.
Overall, this course is putrid.
Usually, there are 1 to 3 items. Frequently, administrators, such as your department chair, associate dean, or emperor or empress, will be encouraged to use the ratings on those items to provide a simple, quick-and-dirty measure of your teaching performance. Those ratings, in lieu of the total scale or subscale scores, are used in conjunction with other information to arrive at summative decisions regarding merit pay, contract renewal for full-time and adjunct faculty, and promotion and tenure recommendations. Are these important decisions about your career and life. You bet!!
Do you want those decisions to be rendered on the basis of 1 or 2 items? “Sure, why not?” Are you kidding me? There are several logical, psychometric, and legal reasons why global items should NOT be used for summative decisions. They will be described in my next blog. Stay tuned for more fun from RatingWorld.
COPYRIGHT © 2009 Ronald A. Berk, LLC
After the last few post-Halloween blogs, you’re probably sick of this topic and the dribbles of info in each blog. Sorry about that. This is the final installment on the “you know what.”
What is the BOTTOMLINE, HANDS-DOWN and UP, BEST STRATEGY TO USE? (Note: You don’t find this type of excitement and suspense in every blog. Brace yourself. Here it comes.) ANSWER: Withhold the posting of final grades. If your registrar is super-efficient at posting grades online in a timely fashion, you’re sunk. That is usually not a problem, because he or she has a bazillion courses to process.
If students are given a 48-hour window following the final exam or project within which to complete the evals, post the grades online on your course Website ASAP. Tell the students in advance both in class and online when they will be posted. That’s the academic version of a “grade (as opposed to movie) trailer” or teaser. This quick post will be perceived as a monster-size carrot or hot fudge sundae to your students. (Note: Certainly there are some courses where students can compute their own final grades, but they need the scores or grades from the final exam or project.)
We can leverage this Net Generation’s characteristics to maximize the response rates we need. Why should they bother to rate our performance? Although there may be some legitimate intrinsic reasons, why take a chance? Go after the extrinsic ones. As noted in my August 2009 blog series, these Net Geners have a need for speed, immediate gratification, and quick feedback on performance, plus they are driven to achieve, feel pressure to succeed, and expect rapid responses from us on everything. Receiving grades “immediately” feeds into these characteristics.
Institutions engaging in this practice over the past few years, including my own, have reported 80–90%+ response rates for most all courses with buckets of typed comments to open-ended or unstructured questions to boot. It might be worth giving it a whirl at your institution.
Let me know your thoughts and experiences with this issue and any other techniques you have used successfully to boost response rates.
COPYRIGHT © 2009 Ronald A. Berk, LLC
Based on the Top 10 List of Strategies in my previous blog, which ones did you pick as MOST EFFECTIVE? Oh, you didn't read my previous blog. Shame, shame. Is this your 1st blog reading? Well, congrats! Welcome to Berk's BlogWorld. I hope you find it informative and fun!
For those of you who did peruse my Top 10, what did you pick? Let’s examine the evidence and see whether you get the prize. (SIDEBAR: “Yo, Blogboy, you didn’t say a prize was involved!” Oops, sorry. It was an afterthought.)
To date, the evidence indicates that students must believe (Do you believe? Kinda like Peter Pan!) that the results will be used for important decisions about faculty and courses in order for the online system to be successful. When faculty assign students in class to complete the evaluation with or without incentives, response rates are high. Finally, withholding early access to grades is supported by students as highly effective, but not too restrictive. This “withholding” approach has been very successful at raising response rates to the 90s at several institutions.
These strategies have boosted response rates into the 80s and 90s. It is a combination of elements, however, as suggested above, that must be executed properly to assure a high rate of students' return on the online investment. The research base and track record of online student ratings belies “low response rate” as an excuse for not implementing such a system in any institution.
While the combo strategy is recommended, there is one BEST technique among the top 10 that can really spike your response rate. It will be examined in the next blog.
Let me know if you have tried other strategies that have worked in your evaluation system.
COPYRIGHT © 2009 Ronald A. Berk, LLC
Here is a list of the Top 10 most effective strategies to get 80%+ response rates for online student ratings (Berk, 2006; Johnson, 2003; Sorenson & Reiner, 2003):
10. Faculty communicate online and face-face to students the importance of their input and how the results will be used
9. Ease of computer access for all students
8. Assurance of anonymity
7. Convenient, user-friendly online system
6. Provide easy-to-understand instructions on how to use the system
5. Administration and faculty strongly encourage students to complete forms
4. Faculty “assign” students to complete forms
3. Faculty threaten to smash students’ iPods with a Gallagher-type sledgehammer
3. System withholds students’ early access to final grades
2. Faculty provide extra credit or points
1. Faculty provide positive incentives (aka assorted bribes, such as requiring students to pick-up dry cleaning and take dog to the vet)
This is quite a laundry list of techniques. Is there any single practice that has a proven track record of success? Yup. That one is even more top-secret and a higher security “code chartreuse” technique than the others. It will be revealed in my next blog. Stay tuned.
COPYRIGHT © 2009 Ronald A. Berk, LLC
There have been cries, yelps, screams, shrieks, screeches, shrills, howls, and other sounds reverberating throughout the academic hallways and byways about low response rates from online administrations of student rating scales. Traditionally, the rates for in-class paper-based administrations following appropriate standardized procedures have been 80%+.
SAMPLING BIAS
First, let’s be Jack Nicholson A Few Good Men “crystal” clear about the problem with low response rates. When ratings dip below 80%, you’re in deeeep trouble! WAIT! That’s not the ending to that sentence. Where did it go? Oh, here it is: sampling bias increases, so that the summarized ratings will present an unrepresentative, biased picture of teaching performance in a given course. Such ratings could be inflated or deflated due to bias. This is evil, especially when student ratings constitute the only source of evidence on teaching performance, which is the case at many institutions. The results may be useless for either formative or summative decisions.
VALIDITY PROBLEM
The intractable problem is that there is no way to detect the direction or degree of bias. This low degree of rating score validity makes it extremely difficult to interpret the ratings based on the limited sample of students who chose (or self-selected themselves) to complete the scales. Keep in mind, these inaccurate ratings may still yield a high degree of internal consistency reliability, which can be misleading.
REASONS FOR NONRESPONSE
The response rate for online administrations can be half the rate or lower of paper-based administrations. This is a frequent objection to online ratings reported in faculty surveys. This fear of low response rate has deterred some institutions from adopting an online system. The research on this topic indicates the following possible reasons for the nonresponses: student apathy, perceived lack of anonymity, inconvenience, inaccessibility, technical problems, time for completion, and perceived lack of importance (Ballantyne, 2000; Dommeyer, Baum & Hanna, 2002; Sorenson & Reiner, 2003).
That’s the problem. Now how do we fix it? Several institutions have tested a variety of strategies to increase response rate that address several of the aforementioned reasons. These strategies have been suggested by faculty AND students. My next blog will present the Top 10 most effective strategies. Stick around as the plot thickens.
COPYRIGHT © 2009 Ronald A. Berk, LLC
