I’m sorry to interrupt my blog series on “How to Put Pizazz into Your PowerPoint Conference Presentations,” but an article was just published that might be useful to some of you.
Are you struggling with issues related to student ratings of teaching, peer evaluation, teaching portfolio, and which gifts to buy for your family for the holidays? Well, have I got a deal for you. I have 2 articles recently published on a faculty evaluation model that might interest you. They are an extension of the model described in my Thirteen Strategies to Measure Teaching book (see Stylus Publishing link in right margin), applied specifically to formative and summative decisions about faculty. The model can be used to evaluate teaching performance and professionalism. Here they are:
Berk, R. A. (2009a). Beyond student ratings: “A whole new world, a new fantastic point of view.” Teaching Excellence, 20(1).
Berk, R. A. (2009f). Using the 360° multisource feedback model to evaluateteaching and professionalism. Medical Teacher, 31, 1073–1080.
The 1st article above is a brief description of the model; the 2nd is a full-blown presentation of the model and lit review. Although the latter was written for professors and administrators in medical schools, all of the characteristics are generalizable to any discipline, department, school, or kingdom. The model is simple, straightforward, and easily applied to impress even accreditation reviewers of your evaluation plan in your self-study. An abstract of the article is given below:
This MT article provides an overview of the salient characteristics, research, and practices of the 360° MSF models in management/industry and clinical medicine. Drawing on that foundation, the model was adapted to the specific decisions rendered to evaluate faculty teaching performance and professional behaviors. What remains unchanged in every application is the original spirit of the model and its primary function:
Multisource Ratings→Quality Feedback→Action Plan to Improve→Improved Performance.
Although the ratings were intended for formative decisions, in many cases they have also ended up being used for summative decisions. All of these applications of the 360° MSF model have advantages and disadvantages. In fact, it is possible to distill several persistent and, perhaps, intractable psychometric issues in executing these models. The top 10 issues are described. Although much has been learned during the 80-year history of scaling, 60-year history of faculty evaluation, and 50-year history of the 360° MSF model in management/industry, a lot of work is still necessary to realize the true meaning of “best practices” in evaluating teaching and professionalism.
You can download the articles from my Website (www.ronberk.com) under Publications. They are intended for your own use and research purposes. Please do not distribute to family members or farm animals. The latter may eat them and get sick. Enjoy!
COPYRIGHT © 2009 Ronald A. Berk, LLC
Recently, I was involved in a spirited marathon discussion with a bunch of colleagues on technical issues related to student ratings of teaching performance. One big topic was: How do you report results for formative and summative decisions? I thought some of my bloggees might be interested in the options available. These options with report form examples appear in my Thirteen Strategies... book (see Stylus link to right).
In order to answer the question, you don't need to administer multiple rating forms. There are a lot of options with the results from just one form. It is possible to "have your cake.." with one form for both formative and summative decisions up to a point. The trick is how the results are analyzed and reported for each decision maker.
Psychometrically speaking, I recommend the following:
1. A structured scale with 4-6 subscales measuring separate constructs such as Class Organization, Teaching Methods, Evaluation Techniques, and so on. The faculty evaluation lit reports several major constructs based on factor analyses. These core teaching behaviors should be generic enough to apply to most courses and disciplines.
2. A separate section devoted to course-specific items each instructor might want to add should be included. This optional section might contain up to 10 items.
3. One to three global items may be included as well, although individual item alpha reliabilities are typically much lower than item aggregates, such as subscale or total scale scores.
4. An unstructured section containing 2-5 stimulus questions to which students can comment is also important. Loads of online administrations reveal students spent considerable time typing buckets of comments. Frequently those comments explain the responses to some of the structured item ratings. Both forms of evaluation are valuable and furnish complementary information on teaching performance.
Analysis-wise, the above structure permits results at the following levels:
1. anchor distribution of percentages
2. item statistics such a mean and median (almost all distributions are negatively skewed)
3. subscale statistics
4. total scale statistics
5. summary of comments by stimulus question
That's a lot of information. Faculty would benefit from 1-5. 1 and 2, in particular, provide valuable diagnostic info to revise teaching or course materials that will benefit their next course-load of students. It is formative feedback only in that sense. Other formative methods administered during the course should be considered. You already know about those options.
Summative decisions by department chairs, associate deans, etc. can be based on 3 and 4 and possibly the global item scores.
The above strategy is certainly not new, but it is the simplest to get the biggest bang from your student rating scale. Of course, it is only 1 of 14 sources of evidence you might use in measuring teaching performance. Multiple sources of evidence should be involved in summative (personnel) decisions about faculty contract renewal, merit pay, and promotion and tenure. After all, faculty careers are on the line.
If you're grappling with this issue, I hope these suggestions may be helpful.
COPYRIGHT © 2009 Ronald A. Berk, LLC
