Recommendation reviewing

What is recommendation reviewing? 

For some programs, merely collecting a numerical score for submissions doesn't quite fit the bill. These accounts typically wish for reviewers to provide accolades for applications that go above and beyond or bring something special to the table. If you'd like reviewers to mark deserving applications as "Recommend to fund", "Gold medal", "Best in show", or something else entirely, recommendation reviewing may be just what you're looking for! 

If your program has implemented our Scoring reviewing mode the instructions below will be quite familiar. For those that haven't, see the steps outlined below to begin configuration. 

Step 1: configure a score set

Recommendation reviewing utilises the framework provided by our Scoring reviewing mode. As such, there is significant overlap in the process starting from the beginning: the creation of a Scoring score set. 

1. In the Manage workspace, navigate to Reviewing > Settings > Score sets
2. Click New score set and choose your desired form from the drop-down
3. Select the Scoring radio button under the 'Reviewing mode' heading
   
4. Assign a Score set name, noting that this name will be shown to your program's reviewers so it should be clear and concise; i.e. 'Stage 1'. 
5. Click Save when finished 

The above steps are the basic requirement for a Scoring score set, but additional options are available. For a breakdown on each of those settings see our dedicated guide: Configure Scoring mode. 

Step 2: configure recommendation scoring criteria

Like with standard Scoring, recommendation utilises scoring criteria. The key difference for recommendation is that, instead of reviewers selecting a numerical value to score the application, they can choose a word or phrase. To configure a recommendation scoring criteria, follow these steps:

1. Open the Manage workspace and go to Reviewing > Settings > Scoring criteria
2. Create a New criterion and choose the desired form from the drop-down
3. Provide a Score title (maximum 32 characters), this is required
4. Input a Shortcode; this is used in summary tables and exports where space is limited— the full score title may be too long and take up too much space
5. The Label (the visible heading associated with the criterion), Help text, and Hint text are all optional
6. If you have multiple Score sets, select the one just created
7. Under the 'Control' heading, choose the Recommendation radio button
8. Two default options (whether or not to recommend to fund) will appear, but they can be edited, reorganised, or added to:
   • To edit a Label or its associated Score, simply click into the field and make your adjustment
   • To add further options, hover your cursor between those that appear or below the last and click the + that appears
   • To reorder existing options click and drag them into your desired sequence
9. Provide a desired Weight for your recommendation criteria 
   
   Tip: if you don't wish for your recommendation criteria to impact the overall scoring of an application, set this value to 0. 
10. Under the 'Commenting' heading, choose your desired option(s)
    
    
    • Comment on individual scores invited will make a comment area on this criteria available to reviewers.
    • Comment on individual scores required will make a comment on this criteria compulsory for reviewers. 
11. If the scoring criteria you're creating should be displayed alongside a specific application field, select that field from the Align this score with a field drop-down, otherwise it is simply positioned at the bottom of the application 
12. Provide an Order for the scoring criteria, this controls the placement of the scoring criteria on screen for the reviewer, in score matrix and exports
    
    Note: if you choose to align the score with a field as mentioned above, there is no need to assign an order value.
13. Finally, chose which Categories (all or some) this scoring criteria should be used for
14. Save when finished

Example: recommendation criteria will appear to reviewers similarly to standard scoring criteria. 

Set up a reviewing round

1. In the Manage workspace, go to Reviewing > Settings > Rounds
2. Click Add round and choose your desired form
3. Select the Reviewing round radio button under 'Round type'
4. Set Starts and Ends dates/times
5. Save

For more detailed instructions, see: Ultimate guide to rounds. 

Create a reviewing panel

Panels help you easily manage which applications go to which reviewers. You can configure as many panels as necessary for a given score set— you might typically have one panel per category. 

1. In the Manage workspace, go to Reviewing > Settings > Panels
2. Create a New panel and choose your desired form from the drop-down
3. Choose Scoring as the reviewing mode, and select the score set created earlier
4. In the 'Applications' section, check the categories/chapters/tags for applications that are to be reviewed
   • There are two additional settings in this section: Only include moderation approved applications and Include archived applications. If moderation is in use, checking the first box will ensure only approved applications go through to reviewing. Checking the second box will include archived applications. 
5. In the Who section select who will be allowed to review whether that be all users with a specific role to review or only selected users
6. In the When section, check the round configured earlier, for when reviewing will be permitted
7. Save

Testing your recommendation configuration

It's important to test your reviewing configuration to ensure that the entire process works in a way that meets your organisations needs and expectations. For a full breakdown on how to test your recommendation reviewing see: How do I test my reviewing setup?

Notes on recommendation

• The scores associated with each recommendation option must be unique meaning that multiple labels cannot hold the same value. 
• If the scores associated with recommendation shouldn't impact the overall score of an application, set the Weight to zero. 
• Reviewers will see the score of each application and any given recommendation in their list view. 
• The reviewing leaderboard can display columns with counts for each recommendation option by clicking the cog icon at the top of the list view. 
• Recommendation results are not included with standard Results - Scoring exports, but can be added as columns in a custom export. For more information see our Ultimate guide to exports.