Peer Review Rating System

The current protocol for Peer Review ratings on MERLOT is to post ratings of 3, 4, or 5 stars and not to post ratings that are 1 or 2 stars. Consequently, when the user sees that no stars are attached to the learning material, this could mean that the material was rated poorly by the peer reviewers or the material has not been reviewed; the user is not able to distinguish between these two possibilities if there are no Peer Reviews.

Note: User Comments do have the 1-5 star range so users can (and have) provided 1 and 2 star ratings. Any MERLOT member and any individual reviewer of the peer review teams can post User Comments if they wish.

Why not post ratings of 1 and 2 stars for Peer Review Reports?

1. MERLOT is an open resource. People contribute material to the collection because they see some value in the material. For an open resource to work, it must enable participation and provide value to the participants. There is a diversity of standards related to quality and this diversity is supported by the principle of academic freedom. Academic freedom does not mean that faculty are free to do a "poor job" teaching but it is a delicate balance between establishing quality standards and enabling the freedom to use diverse learning materials.
   
   
2. MERLOT is a resource of End Products and Developing Products. The End Products should be ready for immediate and effective use. Developing Products are at an early or intermediate stage of development and will by definition have problems. Both End Products and Developing Products provide ideas and guidelines for developers and users of instructional technology. It is critical to support the Developing Products because we need to create a process for encouraging the expansion of the collection.
   
   
3. There are a lot of good learning materials "out there" to review first. Practically speaking, identifying the good materials first is more important and useful than identifying the bad materials first.
   
   
4. MERLOT needs to build a sustainable culture of volunteerism and professional responsibility.

   a.  Posting positive reviews supports the good materials and provides  recognized samples of good materials to the community.
   b.  Constructive criticism of the "bad" materials will identify areas that need improvement. The feedback can be used to engage authors to participate more fully in MERLOT. There are several scenarios that may result from this:

1. Author can remove the materials from MERLOT.
2. Author can improve the materials and resubmit to MERLOT.
3. Author can choose not to have the review posted and leave the material in MERLOT. Non-rated materials are initially sorted to the bottom of the list. As more positive reviews are generated, the non-rated materials will require more "active searching" by users.
4. Reviewers can provide User Comments that could identify areas that need improvement. User Comments can be rated 1-5.
5. Materials that have limited value could have a slight, but useful purpose for a small group of faculty and students.