Elements Assessment Module Guide

The REF Manager Publications Report is designed to give Unit Managers a summary of the output selections in an exercise. 

Summary 

The Summary tab/sheet gives an overview of the Exercise Definition in the form of a list of the unit(s) you have been assigned to manage. It shows counts of the number of assigned users, publication selections made, and distinct outputs in play for each unit, and the 'Number of required REF2 outputs' for each Unit. There's one row on the summary for each Unit of Assessment.

The Summary tab also gives visibility of the number of REF2 records in each unit, broken down by Submission status (Selected, Accepted, Attributed, Not attributable, and Not accepted).  Within a given a unit of assessment, for any given output - identified by Publication ID - only one REF2 is permitted, so the total number of REF2 Submission statuses will be equal to the distinct number of outputs in a unit.  

In contrast to other REF stock reports, each output is counted once in this tab.  No account is made for double-weighting or double-attribution. 

Outputs 

The Outputs tab/sheet contains one row per user selection. Each row shows:

- the Unit, List Name, user details, Publication ID
- details of the selected output including title, and type
- reviewer scores and comments from up to the first four reviewers
- the REF2 Submission status for each selection
- Web of Science citation counts (because of the contract to supply data to the Research Councils)
- the output's DOI, journal title (blank if not relevant), and a list of other authors

Any review comments and scores are shown, up to the fourth reviewer for a given output, in the order they were added, irrespective of review stage in which the review was created, and irrespective of whether the reviewer is still assigned as a reviewer of the selecting user's outputs in the current stage, or the stage at which that review was originally added. 

Identifying Publications Selected Multiple Times

We have included visibility of the number of times each output has been selected in the unit, as well as a filterable column to enable highlighting of selections with an “Invalid Multiple Selection Flag”. Objects can only be invalid selections in this way if they are currently selected by the same user more than once in the same list, as a consequence of joining / merging two objects which had both been selected, and this report provides administrators/managers a quick way of finding occurences of this.

The REF Reviewer Score Calibration Report provides information on how individual reviewers have been scoring. It can be accessed by Managers via the "Export unit" button within the management area of the exercise. 

The report renders to Excel and comprises a cover sheet block and a table of reviewers. For each reviewer, it shows a single row with:

- the number of item scores they have supplied
- the minimum, maximum, average and standard deviation of their numerical score values
- a textual representation of the distribution of their numerical score values, showing each individual score in ascending order, separated by space characters.

All review scores are shown in this report, irrespective of review stage in which the review was created, and irrespective of whether reviewer is still assigned as a reviewer of the selecting user's outputs in the current stage, or the stage at which that review was originally added. 

The REF2 Outputs Report is designed to give managers of a unit access to a detailed list of all REF2 Research Outputs in their output pool.

The report is available in Excel format and has two tabs/sheets: 

- Outputs
- Affiliations

Outputs tab

The 'Outputs' tab lists all REF2 outputs and their associated metadata.  This includes:

- Submission status
- Attributed REF1 name(s) and details
- Category and Research Group(s) of Attributed REF1 Researcher(s)
- REF2 counts at the Unit and REF1 Researcher levels
- Overall predicted score and notes
- Publication year and month (derived values and overrides)
- REF2 Output type
- Open Access details (for D-Journal articles and E-Conference contributions only):
          -- Relevant values from the Open Access Monitor (Acceptance date, First deposit, Publication date)
          -- REF OA status
- REF2 metadata fields such as Additional information, Interdisciplinary, Forensic, Cross-referral, Sensitive
- Bibliographic metadata such as Place, Publisher, Article number, etc.
- WoS ID and citation count (in relevant UoAs)
- Output allocation (in relevant UoAs)
- Double-weighting and reserve
- Pending and reserve (while the field is no longer to be used, it still exists in the datamodel)
- English abstract
- COVID delayed output flag and affected output statement
- Electronic / Physical output submission management information
- REF2 Data confirmation workflow

Where a REF1 Researcher name is followed by an asterisk in the report, this indicates that the researcher selected the item for the exercise.  

Affiliations tab

The purpose of the 'Affiliations' tab is to help with attribution decisions by listing all REF2 outputs, one row for each combination of REF2 and affiliated REF1 Researcher.  Fields available include:

- Submission status
- Affiliated REF1 name(s) and details, including Category and Research Group(s)
- REF1 author contribution details
- Attributed REF1 name(s) and details, including Category and Research Group(s)
- REF2 counts at the Unit and REF1 Researcher levels 
- REF2 Output type
- Overall predicted score and notes
- Publication year and month
- REF OA status
- Double-weighting

Where a REF1 Researcher name is followed by an asterisk in the report, this indicates that the researcher selected the item for the exercise. 

**This report will likely be most used by central teams rather than UoAs. 

The REF2 Modified publications report lists all REF2 Outputs where the associated publication within Elements has been most recently modified (by merging / splitting) during the selected date range, and is designed to help you see when underlying publication data may have changed due to merging, splitting, or new records being created within the publication.

Notes:

- Report includes only REF2 outputs in the following submission states:
          -- Accepted
          -- Attributed
          -- Not attributable
- Date range selection:
          -- The report only captures the most recent modification (merge / split). So, if you select a date range of 1 May 2020 - 15 May 2020, and the publication was modified on 12 May and 17 May, the output will not be listed as the most recent modification (17 May) was outwith the selected date range.
- There may be multiple REF2s associated with a single publication - so both affected REF2s would be listed (depending on submission state)
- Where the report is notifying about a new record being created in a publication, this is any new record - not just those where a material difference has been made to the REF2
- This report may not immediately reflect recent changes to data

**This report will likely be most used by central teams rather than UoAs. 

The REF Submission validation report is designed to give managers of a unit or an exercise definition ready access in Microsoft Excel to a detailed list of all REF validation errors and warnings.

Note that the REF Submission Validation report validates your REF content against the rules outlined in the REF2021 submission system validation rules.  There may be additional undocumented rules applied in the REF Submission System.  Accordingly, we recommend that you also ensure you run validation reports directly in the REF Submission System. 

This dashboard provides a summary of the REF1 and REF2 elements of the selected Unit's potential REF2029 submission. Includes Cat A FTE, Cat A and B headcounts, and a REF2 attribution summary including tracking the number of attributed REF2s against the number required. (Please note, FTE/headcount figures will not yet be correct in Elements. To view accurate information, please access the Research Dashboard). 

REF1 Researchers

CHART: Included in submission : Headcount % Cat A v Cat B

This chart shows the composition of the submission, displaying the proportion of Cat A v Cat B headcount.

TABLE: REF1 Summary: Included in Submission

This table presents summary-level information relating to staff flagged as included in the submission.

TABLE: REF1 Summary: Not included in submission

This table presents summary-level information relating to staff flagged as NOT included in the submission.

REF2 Output attributions

CHART: Progress towards reaching required number of REF2 attributions

This chart shows the progress towards reaching the required number of REF2 attributions, showing the current number of attributed REF2s against the number required.

TABLE: REF2 Attribution Summary

This table presents summary-level information relating to REF2 attributions.

TABLE: Additional REF2 requirement variables taken into account

This table presents additional variables that are taken into account when calculating the number of required REF2s and the number of Attributed REF2s.

This dashboard provides a number of indicative quality profiles for REF2 Outputs, and is separated into two tabs:

- Attributed and Accepted REF2s
- Cat A v Cat B

Attributed and Accepted REF2s tab

This tab provides three separate indicative quality profiles (presented both as tables and graphically), designed to help you ensure that the quality of your Attributed REF2 profile is maximised:

- Attributed REF2s
- Attributed + Accepted REF2s
- Accepted REF2s

Each profile provides the following:

- The number and proportion of REF2s at each quality level (as captured in the overall predicted score on the REF2 form) (excludes Reserves) 
- The number of 'No score' REF2s, which includes:
          -- REF2 outputs with no overall predicted score recorded
          -- REF2 outputs to be input directly into the REF submission system 
- GPA (Grade Point Average) (excludes any 'No score' REF2 outputs)

Note that Attributed REF2s include any outputs attributed to REF1 Researchers NOT included in the submission (including any REF2 outputs to be input directly into the REF submission system (as captured in the REF1 form of the NOT included researcher)).  It is recommended that any such attributions are removed to ensure accurate calculations.

Cat A v Cat B tab

This tab provides three separate indicative quality profiles (presented both as tables and graphically), designed to help you ensure that Cat B staff included in the submission serve to raise the overall quality of the submission:

- Attributed REF2s
- Attributed REF2s : Cat A
- Attributed REF2s : Cat B

Each profile provides the following:

- The number and proportion of REF2s at each quality level (as captured in the overall predicted score on the REF2 form) (excludes Reserves)
- The number of 'No score' (not graded) REF2s, which includes:
          -- REF2 outputs with no overall predicted score recorded
          -- REF2 outputs to be input directly into the REF submission system (as captured in the REF1 form)
- GPA (Grade Point Average) (excludes any 'No score' REF2 outputs)

Note that these profiles include any outputs attributed to REF1 Researchers NOT included in the submission (including any REF2 outputs to be input directly into the REF submission system (as captured in the REF1 form of the NOT included researcher)).  It is recommended that any such attributions are removed to ensure accurate calculations.

This dashboard provides information on the distribution of attributions across REF1 Researchers, and is separated into two tabs:

- REF2 Attribution profile
- Research Group profile

REF2 Attribution profile tab

This tab provides three separate tables, showing the distribution of number of REF2 attributions across REF1 Researchers:

- REF2 attributions of REF1 Researchers included in the submission
- REF2 attributions of Cat A staff included in the submission
- REF2 attributions of Cat B staff included in the submission

Each table shows the number and proportion of included REF1 Researchers with:

- 0 attributions (and no REF6a waiver) (highlighted in RED if this value is not zero)
- 0 attributions (with a REF6a waiver)
- 1 attribution
- 2 attributions
- 3 attributions
- 4 attributions
- 5 attributions
- more than 5 attributions (highlighted in RED if this value is not zero)

This tab also includes a pie chart which is a graphical representations of the first table (Proportion of REF1 Researchers by number of REF2 attributions).

Research Group profile tab

This tab compares the Research Group assignment distribution of REF2 attributions with the Research Group assignment distribution of REF1 Researchers. 

The REF2 attributions table shows the number and proportion of attributed REF2s assigned to each Research Group in the Unit, including any with no RG assignment (highlighted in RED if this value is not zero) and any with no RG assignment as they are being input directly into the REF submission system). Includes any outputs attributed to REF1 Researchers NOT included in the submission (which it is recommended are removed to ensure accurate calculations)). Excludes Reserves. Note that where a REF2 is double-attributed and one of the attributed REF1 Researchers does not have a RG assignment, this may result in an odd count here. It is best to resolve the issue by ensuring that both attributed REF1 Researchers have RG assignments.

The Cat A FTE of REF1s included in the submission table shows the FTE and proportion of included REF1 Researchers assigned to each Research Group in the Unit, including any with no RG assignment (highlighted in RED if this value is not zero). Note that where a REF1 Researcher is assigned to multiple Research Groups, their FTE is split evenly across the RGs.

The two charts are visual representations of these two tables.

This dashboard provides details of REF2 properties, and is separated into three tabs:

- REF2 Open Access (following the REF2021 Open Access policy as the REF2029 policy has not been released)
- REF2 publication properties
- REF2 properties

REF2 Open Access tab

This tab provides three separate REF2 Open Access profiles, designed to help you ensure that you do not exceed the 5% 'Not compliant' threshold in your submission:

- Attributed REF2s
- Attributed + Accepted REF2s
- Accepted REF2s

Each profile provides the number and proportion of REF2s with each REF2 Open Access status.

Notes:

- These tables only capture D-Journal articles and E-Conference contributions
- Proportion calculations exclude 'Out of scope' outputs
- Where the number of 'Not compliant' outputs exceeds 5% of the submission, this is highlighted in RED (note that as the 5% tolerance in the submission itself is calculated by rounding to the nearest whole output number, we do not do any such rounding in this dashboard and simply flag up any value exceeding 5%)
- Reserves are excluded
- Outputs to be input directly into the REF submission system are not included
- Attributed REF2 counts include any outputs attributed to REF1 Researchers NOT included in the submission.  It is recommended that any such attributions are removed to ensure accurate calculations.

REF2 publication properties tab

REF2 Output Type

This table and chart show the number and proportion of REF2 attributions of each REF2 Output type (as captured on the REF2 form).

Notes:

- Where there are any attributed outputs where the REF2 Output type is not set, these are highlighted in RED
- Reserves are excluded
- Outputs to be input directly into the REF submission system are not included (and this includes when calculating the proportions)
- Attributed REF2 counts include any outputs attributed to REF1 Researchers NOT included in the submission.  It is recommended that any such attributions are removed to ensure accurate calculations.

REF2 Publication Year

This table and chart show the number and proportion of REF2 attributions published in each year (as captured on the REF2 form).

Notes:

- Where there are any attributed outputs where the Publication year is as follows, these are highlighted in RED:
          -- ≤ 2013
          -- ≥ 2021
          -- No publication date
- Reserves are excluded
- Outputs to be input directly into the REF submission system are not included (and this includes when calculating the proportions)
- Attributed REF2 counts include any outputs attributed to REF1 Researchers NOT included in the submission.  It is recommended that any such attributions are removed to ensure accurate calculations.

REF2 properties tab

REF2 Properties

This table and chart show the number and proportion of REF2 attributions with various REF2 properties (as captured on the REF2 form).

Notes:

 - As each property is not exclusive, it is possible for a single output to have a number of the listed properties.  
- Reserves are excluded 
- Outputs to be input directly into the REF submission system are not included (and this includes when calculating the proportions)
- Attributed REF2 counts include any outputs attributed to REF1 Researchers NOT included in the submission.  It is recommended that any such attributions are removed to ensure accurate calculations.

Multiple author attribution

This table and chart show the number and proportion of REF2 attributions that are attributed to two REF1 Researchers (as captured on the REF2 form).

Notes:

- In Panel D, where the number of single-weighted double-attributions is greater than 5% of the submission, this is highlighted in RED
- Reserves are excluded
- Outputs to be input directly into the REF submission system are not included (and this includes when calculating the proportions)
- Attributed REF2 counts include any outputs attributed to REF1 Researchers NOT included in the submission.  It is recommended that any such attributions are removed to ensure accurate calculations.

This dashboard highlights data consistency issues across the REF submission in the selected Unit. It does not specifically include REF submission validation (see separate REF Validation registered report) (although there will be some overlap).

Colour legend:

- GREEN = No action required
- RED = Action required
- ORANGE = Action potentially required

Attributed and Accepted REF2s tab

REF1 Researchers included in the submission table

REF1 Researchers (includes those flagged as NOT included in submission

REF2 Research Outputs

Notes:

- Reserves are excluded
- Outputs to be input directly into the REF submission system are not included (except in the REF2 attribution shortfall calculation)
- Attributed REF2 counts include any outputs attributed to REF1 Researchers NOT included in the submission.  It is recommended that any such attributions are removed to ensure accurate calculations.

REF2 attribution shortfall

Where [Number of Attributed REF2s] + [Outputs being input directly in REF submission system] is ≠ [Number of required REF2s], the cell is highlighted in RED. The number is presented as a negative if there are less REF2 attributions than are required;  the number is presented as a positive if there are more REF2 attributions than are required.