About the Peer Review Process

The Peer Review Process for evaluating the teaching and learning resources in MERLOT follows the model of the peer review of scholarship.

Faculty selected in accordance with MERLOT guidelines perform the peer review of MERLOT learning resources. The peer review process is led by Editors and an Editorial Board and includes:

1. Developing Evaluation Standards
   • The MERLOT Editorial Boards provide leadership, tools, and training in developing evaluation standards and processes.
   • The MERLOT Administrative Team provides the Editorial Boards with a framework of evaluation criteria which is based on:
     • Quality of Content
     • Potential Effectiveness as a Teaching Tool
     • Ease of Use
   • All Peer Reviewers on each Discipline-specific Editorial Boards share share and compare their evaluations following the processes developed and the framework provided to create test cases. These test cases are then used to develop evaluation guidelines/criteria that are applied to all materials in the discipline.
   • Each Editorial Board establishes substantial inter-rater reliability in its evaluations before evaluation procedures are implemented on the remaining materials in its discipline.
     
     
2. Conducting Evaluations:
   • Editorial Boards decide on the process for selecting materials to be peer reviewed.
   • If a collection of materials is not large enough to adequately establish inter-rater reliability, its Editorial Board works to expand the collection sufficiently.
   • Review teams typically use the following two stage review process:
     Stage 1 Cursory Review to Identify Worthy Candidates:
     • The Editorial Board reviews its collection and "triages" materials as follows:
       1. Definitely worth reviewing
       2. Possibly worth reviewing
       3. Not worth reviewing at this time
     • Peer Reviewers and Associate Editors report their cursory evaluation to their Editorial Boards.
     • Peer Reviewers can post Member Comments based on cursory evaluations as appropriate.
     • The Editorial Board compiles its list of worthy materials. Those deemed "definitely worth reviewing" receive top priority in the review process.

     Stage 2 Intensive Review of Worthy Candidates:

     • The Editorial Board assigns "worthy" materials to Editorial Board Members.
     • In some cases, the Editorial Board asks the author(s) of the material(s) for permission to review.
     • Two Peer Reviewers each use their Editorial Board's review procedures, forms, and evaluation standards as they independently review the material.
     • Reviewers write peer review reports using the evaluation criteria for MERLOT learning materials as a guideline and publish these individual reviews in their Editorial Board Workspace.
     • If there is any significant disparity in the two reviews, an Editor or Associate Editor assigns the material to a third reviewer.
       
       
3. Reporting Evaluations:
   • The Editor or Associate Editor reviews both individual reviews and creates an integrated or Composite Peer Review Report on Workspace.
   • The "author" of a Composite Peer Review Report is listed as "MERLOT [Discipline] Review Panel".
   • A list of the members of the "MERLOT [Discipline] Review Panel" is posted on MERLOT.
   • The Editorial Board sends the Composite Peer Review Report to the author(s) for feedback and permission to post the review on MERLOT.
   • The Editorial Board posts the Composite Peer Review Report on the MERLOT website.
   • Authors may ask the Editor to send two letters, to two individuals of their choice, summarizing the peer review process and including the Composite Peer Review Report.

For more information please contact Cathy Swift at cswift@calstate.edu