MERLOT Peer Review Process
The MERLOT Peer Review Process for evaluating the teaching and learning resources in MERLOT follows the model of the peer review of scholarship.
Faculty selected in accordance with MERLOT guidelines perform the Peer Review Process of MERLOT on learning resources. The Peer Review Process is led by Editors and includes Editorial Board Members and Peer Reviewers. There are three stages to the Peer Review Process:
Stage 1. Developing Evaluation Standards
The MERLOT Editorial Boards provide leadership, tools, and training in developing evaluation standards and processes. The MERLOT Management Team provides the Editorial Boards with a framework of evaluation criteria which is based on:
- Quality of Content
- Potential Effectiveness as a Teaching Tool
- Ease of Use
All Peer Reviewers on each Discipline-specific Editorial Board share and compare their evaluations following the processes developed and the framework provided to create test cases. These test cases are then used to develop evaluation guidelines/criteria that are applied to all materials in the discipline. Each Editorial Board establishes substantial inter-rater reliability in its evaluations before evaluation procedures are implemented on the remaining materials in its disciplines.
Stage 2. Conducting Evaluations:
Editorial Boards decide on the process for selecting materials to be peer reviewed. If a collection of materials is not large enough to adequately establish inter-rater reliability, its Editorial Board works to expand the collection sufficiently. Peer Reviewer teams typically use the following two step Peer Review Process:
Step 1 Cursory Review to Identify Worthy Candidates:
The Editorial Board reviews its collection and "triages" materials as follows:
- Definitely worth reviewing
- Possibly worth reviewing
- Not worth reviewing at this time
Peer Reviewers and Associate Editors report their cursory evaluation to their Editorial Boards. Peer Reviewers can post Member Comments based on cursory evaluations as appropriate. The Editorial Board compiles its list of worthy materials. Those deemed "definitely worth reviewing" receive top priority in the review process.
Step 2 Intensive Review of Worthy Candidates:
The Editorial Board assigns "worthy" materials to Editorial Board Members. At least two Peer Reviewers use their Editorial Board's review procedures, forms, and evaluation standards as they independently review the material. Reviewers write Peer Review Reports using the evaluation criteria for MERLOT learning materials as a guideline and publish these Individual Reviews in their Editorial Board Workspace (an internal website to MERLOT). If there is any significant disparity in the two reviews, an Editor or Associate Editor assigns the material to a third reviewer.
Stage 3. Reporting Evaluations:
The Editor or Associate Editor reviews both Individual Reviews and creates an integrated or Composite Peer Review Report on Workspace. The "author" of a Composite Peer Review Report is listed as "MERLOT [Discipline] Review Panel". The Editor sends the Composite Peer Review Report to the author(s) for feedback and permission to post the review on MERLOT. When permission is obtained, the Editor posts the Composite Peer Review Report on the MERLOT website. Authors may ask the Editor to send letters, to two individuals of their choice, summarizing the peer review process and including the Composite Peer Review Report.
For more information please contact Cathy Swift
Revised 6/03/08 - CS
|
