Validation for RateMyProfessors.com?

You've heard the reasons why professors don't trust RateMyProfessors.com, the Web site to which students flock. Students who don't do the work have equal say with those who do. The best way to get good ratings is to be relatively easy on grades, good looking or both, and so forth. But what if the much derided Web site's rankings have a high correlation with markers that are more widely accepted as measures of faculty performance? Last year, a scholarly study found a high correlation between RateMyProfessors.com and a university's own system of student evaluations. Now, a new study is finding a high correlation between RateMyProfessors and a student evaluation system used nationally.

A new study is about to appear in the journal Assessment & Evaluation in Higher Education and it will argue that there are similarities in the rankings in RateMyProfessors.com and IDEA, a student evaluation system used at about 275 colleges nationally and run by a nonprofit group affiliated with Kansas State University.

What is notable is that while RateMyProfessors.com gives power to students, IDEA gives a lot of control over the process to faculty members. Professors identify the teaching objectives that are important to the class, and those are the measures that count the most. In addition, weighting is used so that adjustments are made for factors beyond professors' control, such as class size, student work habits and so forth - all variables that RateMyProfessors doesn't really account for (or try to account for). The study looked at the rankings of 126 professors at Lander University, in South Carolina, and compared the two ratings systems. The findings:

* Student rankings on the ease of courses were consistent in both systems and correlated with grades.

* Professors' rankings for "clarity" and "helpfulness" on RateMyProfessors.com correlated with overall rankings for course excellence on IDEA.

* The similarities were such that, the journal article says, they offer "preliminary support for the validity of the evaluations on RateMyProfessors.com."

The study was conducted by Michael E. Stonntag, who formerly taught at Lander and who is now vice president for academic affairs at the University of Maine at Presque Isle, and by two psychology professors at Lander, Jonathan F. Bassett and Timothy Snyder.

Sonntag said that there are two ways to read the results: One is to say that RateMyProfessors.com is as good as an educationally devised system and the other would be to say that the latter is as poor as the former. But either way, he suggested, it should give pause to critics to know that the students' Web site "does correlate with a respected tool."

William H. Pallett, president of IDEA, said he was "surprised a bit" by the correlation between his organization's rankings and those of RateMyProfessors.com. That's because much of the criticism he has heard of the student oriented site is that rankings aren't representative, while much of the effort at IDEA is based on assuring representative samples. "I am surprised, given that we do attend to issues of reliability and validity and they acknowledge that they don't," he said.

Pallett cautioned, however, that IDEA is not intended to be a sole basis for evaluating a course or professor. He said that he would always advise departments to have professors evaluate on another, and to use student evaluations as just one part of that review.

Sonntag said that his current institution uses a home-grown student evaluation system, and that he has no plans to seek a change to IDEA or RateMyProfessors.com - and that the evaluation system is covered by a collective bargaining contract anyway. But he said that he hoped the study might prompt some to think about the online rankings in new ways.

For his part, Sonntag acknowledged that some RateMyProfessors.com reviews are "so mean-spirited" that they aren't worth anyone's time. But he said that if you cast those aside, there are valuable lessons to be learned. He said that he does check what the site says about his teaching - and has found reinforcement for some innovations and reason to question whether some of his tests were too difficult.

"I've been an instructor for 10 years. I look at it," he said, adding that he has found insights "that weren't on my teaching evaluations and I have thought: `Wow. I believe what the student has said is valid and perhaps I can change the way I teach."

Source





Twenty-Five Years Later, A Nation Still at Risk

Today marks the 25th anniversary of "A Nation at Risk," the influential Reagan-era report by a blue-ribbon panel that alerted Americans to the weak performance of our education system. The report warned of a "rising tide of mediocrity that threatens our very future as a nation and a people." That dire forecast set off a quarter century of education reform that's yielded worthy changes - yet still not the achievement gains we need to turn back the tide of mediocrity.

After decades of furthering educational "equality," the 1983 commission admonished the country, it was time to attend to academic excellence and school results. Educators didn't want to hear this and a generation later many still don't. Our ponderous public-school system resists change. Teachers don't like criticism and are loath to be judged by pupil performance. In educator circles, one still encounters grumbling that "A Nation at Risk" lodged a bum rap.

Others heeded the alarm, though, and that report launched an era of forceful innovation and accountability guided by noneducators - elected officials, business leaders and philanthropists. Such "civilian" leadership has brought about two profound shifts that the professionals, left to their own devices, would never have allowed. Today, instead of judging schools by their services, resources or fairness, we track their progress against preset academic standards - and hold them to account for those results. We're also far more open to charter schools, vouchers, virtual schools, home schooling. And we no longer suppose kids must attend the campus nearest home. A majority of U.S. students now study either in bona fide "schools of choice," or in neighborhood schools their parents chose with a realtor's help.

Those are historic changes indeed - most of today's education debates deal with the complexities of carrying them out. Yet our school results haven't appreciably improved, whether one looks at test scores or graduation rates. Sure, there are up and down blips in the data, but no big and lasting changes in performance, even though we're also spending tons more money. (In constant dollars, per-pupil spending in 1983 was 56% of today's.)

And just as "A Nation at Risk" warned, other countries are beginning to eat our education lunch. While our outcomes remain flat, theirs rise. Half a dozen nations now surpass our high-school and college graduation rates. International tests find young Americans scoring in the middle of the pack.

What to do now? It's no time to ease the push for a major K-12 education make-over - or to settle (as Barack Obama and Hillary Clinton apparently would) for reviving yesterday's faith in still more spending and greater trust in educators. But we can distill four key lessons:

First, don't expect Uncle Sam to manage the reform process. Not only does Washington lack the capacity to revamp thousands of schools and create alternatives for millions of kids, but viewing education reform as a federal obligation lets others off the hook. Yet some things are best done nationally - notably creating uniform standards and tests in place of today's patchwork of uneven expectations and noncomparable assessments. These we have foolishly resisted.

Second, retain civilian control but push for more continuity. Governors and mayors remain indispensable leaders on the ground - but the instant they leave office, the system tries to revert. The adult interests that rule it - teacher unions, yes, but also colleges of education, textbook publishers and more - look after themselves and fend off change. If three consecutive governors or mayors hew to the same agenda, those reforms are more apt to endure.

Third, don't bother seeking one grand innovation. Education reform is not about silver bullets. But huge gains can be made by schools that are free to run (and staff) themselves, attended by choice, expected to meet high standards, and accountable for their results.

Consider the more than 50 schools in the acclaimed Knowledge Is Power Program (KIPP) network. We don't have nearly enough today, but we're likelier to grow more of them outside the traditional system than by trying to alter the system itself.

Finally, content matters. Getting the structures, rules and incentives right is only half the battle. The other half is sound curriculum and effective instruction. If we can't place enough expert educators in our classrooms, we can use technology to amplify the best of them across the state or nation. Kids no longer need to sit in school to be well educated.

Far from delivering an undeserved insult to a well-functioning system, the authors of "A Nation at Risk" were clear-eyed about that system's failings, and prescient about the challenges these posed to America's future. Now that we're well into that future, we owe them a vote of thanks. But our most solemn responsibility is to keep the reform flag flying high in the wind that they created.

Source