Configure Qualifying mode

The Qualifying reviewing mode requires a pass or fail decision. This mode might ordinarily be used as a first stage in a reviewing process to eliminate applications that do not meet the base requirements for the category/program. Qualifying might therefore be done by a group of administrators to ensure a preliminary degree of quality and integrity amongst applications that then pass through to a scoring panel of reviewers.

Setting up Qualifying requires three things to be configured:

1. A reviewing round to control access period
2. A qualifying score set to record decisions
3. Panel(s) to assign who can qualify which applications

To configure a Qualifying reviewing round, you can follow the steps below or simply utilise the reviewing fast start. 

Set up a reviewing round

1. Go to Settings > Reviewing > Rounds
2. Create a new round, setting the type to Reviewing
3. Set start and end dates for the qualifying period
4. Save

For more detailed instructions, see: Round configuration.

Create a qualifying score set

1. Go to Settings > Reviewing > Score sets
2. Create a new score set, setting the reviewing mode to Qualifying
3. Qualifying configuration options include:
   • Quantity of responses required for consensus
     For a consensus to be reached, this number of responses must be supplied. For example, it doesn't matter if there are ten reviewers or one hundred reviewers, or if they all say pass or all say fail or a mix of both. The only requirement is that ten responses have been received.
   • % of pass responses required to qualify
     Once a minimum quantity of responses is received, this % controls whether how many reviewers must to agree on a pass to qualify, or just some. For example, if ten reviewers respond and the percentage required to qualify is 60%, six of those reviewers must give a pass. If 27 reviewers respond, 16 of them must give a pass. If one hundred reviewers respond, 60 must give a pass. The algorithm will always round up.
     
     Note: the two rules are not interdependent. The first must be met. If it is, then the second must be met. If Allow unsure decisions has been selected, those responses will not be counted in the algorithm.
   • Allow unsure decisions
     If a reviewer selects this option, it does not count towards the decision.
4. Under the Display tab you have the ability to set other options such as:
   • Category quick filter: to enable reviewers to filter by a category quickly
   • Order applications: choose the order in which you want to display applications for reviewers
   • List length default: choose the number of applications your reviewers see in their reviewing list view. This setting makes it easier for reviewers who have a lot of applications to view everything at once. Options include 10, 25, 50, and 100 applications per page.
   • Commenting options: choose if reviewers can leave a comment on the overall application or individual scores and if the comments are mandatory. You can also select if reviewers should be able to see each other's comments
   • Attachment options: choose if attachments should be displayed and/or available for download
   • Share scores: Control whether you display scores from another score set to reviewers on this score set. See more about displaying score matrixes to reviewers.
   • Slideshow and Display: choose if you want to create a slideshow for all media in the application and choose to show/hide application and applicant name. You can also choose to include scoring boxes on the reviewer PDF.
5. Click Save when finished

Create reviewing panels

1. Go to Settings > Reviewing > Panels
2. Create a New panel
3. Choose Qualifying as the reviewing mode, and select the score set created earlier
4. In the Applications section, check the categories/chapters/tags for applications that can be qualified.
5. In the Who section select who will be allowed to qualify applications.
6. In the When section, check the round configured earlier, for when qualifying will be permitted.
7. Click Save.

Wrapping up

Once all the above is done, refresh the page and if you’re on a qualifying panel and the qualifying reviewing round is currently open, you will see an Applications > Qualify applications link in the navigation.