REF output FAQs

FAQs relating to the outputs component of REF2021.

I have not produced any eligible outputs during the publicaton period - what happens now?

Depending on your cirumstances (including, but not limited to, family-related leave or illness), the School may request that the minimum of one output per member of staff is waived (GoS 158-160; 179-180). The process for making such requests is anticipated to start in Autumn 2019, and the deadline is March 2020 (GoS 198). Further details on the local process will be available once the University's code of practice is finalised, expected June 2019. If you have any questions, please contact Steve Renals ( as the REF co-ordinator to confidentially discuss your case.

Email Steve

Guidance on submissions (REF 2019/01) from the REF2021 website

UoE REF2021 intranet including updates on the development of the code of practice

I want to put forward an output which has not met the requirements of the REF open access policy - what can I do?

A small number of in-scope outputs may be submitted to REF2021 without having met the open access requirements, or having an exception applied. The allowed margin is 5% of the unit's in scope outputs, or 1, whichever is larger (GoS 231). For Informatics, this number is likely to be around 15, which should be put in the context of 170+ submitting staff. 

First review the REF open access policy page and ensure your output is in scope of the policy. If this is the case, and no exception has or can be applied, please contact the RDM team to flag that your output may need to be proposed as one of the allowed non-compliant entries. 

REF Open Access Policy

Contact the RDM team

Guidance on submissions (REF 2019/01) from the REF2021 website

What is the latest an output can be published and still be eligible for REF?

Outputs made publicly available by 31 December 2020 are eligible for submission to REF. The REF team acknowledges that an output may be accepted before the submission deadline (27 November 2020) without certainty of publication prior to 31 December 2020. If such an output is nominated, then a reserve output may also be nominated, which will be evaluated should the first output ultimately not be publicly available by the cut-off date. Full details of the nominated output (including publication date) must be submitted to the REF team by 29 January 2021 (GoS 256.b).

The guidance notes that if the version first made publicly available is the accepted manuscript, then this date or the date the version of record is made available may be used to determine REF eligibility of the output (GoS 257). In real terms, the accepted manuscript is sometimes made available by the publisher before the version of record is complete (i.e. with full typesetting). It is unlikely that accepted manuscripts can be made available via a non-publisher website (e.g. the institutional repository) before the version of record is published. 

Guidance on submissions (REF 2019/01) from the REF2021 website

Can I nominate a conference paper which is being presented in 2020, but with proceedings not being published until 2021?

Probably not. To be submitted as a conference paper, the output needs to be made available as a full paper (e.g. on the conference website or in proceedings) on or before 31 Dec 2020 (GoS 256). Strictly speaking, presentation slides made available prior to the proceedings can be returned (if publicly available within the REF period), but the signifiance of such an output would need to be considered. 

Guidance on submissions (REF 2019/01) from the REF2021 website

Can I nominate papers I authored prior to working at Informatics? 

Yes. Papers authored and published prior to your start with the School are still eligible (GoS 205.c.i). If you would like to confirm the eligibility of a specific output, you are welcome to contact the RDM team.

Guidance on submissions (REF 2019/01) from the REF2021 website

Contact the RDM team

Do my outputs published prior to my start with the School have to meet open access requirements?

No. When the papers are added to Pure by the School's RDM team, an exception is applied to make clear that you were not employed by the School at the time of acceptance, and in turn that open access requirements should not apply (GoS 252.c; 254.a).

Guidance on submissions (REF 2019/01) from the REF2021 website

My outputs do not have huge numbers of citations - will this be a disadvantage?

No. It is not recommended that citation data is used to inform the selection of outputs for submission to REF2021 (GoS 292).

While the Informatics sub-panel will use the citation data provided by the REF team in their assessment of an output, it is acknowledged that citation data are not always a reliable indicator of significance and will not be used as a primary assessment tool (PCWM 274-275, 278; GoS 288-289). 

Guidance on submissions (REF 2019/01) from the REF2021 website

Panel criteria and working methods (REF 2019/02) from the REF2021 website

My proposed output has a huge number of citations - should I include this as additional information?

No. Submissions should not include citation data as this will be provided to panels by the REF team (PCWM 250).

Panel criteria and working methods (REF 2019/02) from the REF2021 website

My outputs are not published in journals with high impact factors - will this be a disadvantage?

No. Sub-panels will not use journal impact factors to evaluate submitted outputs (PCWM 207; 250).

Panel criteria and working methods (REF 2019/02) from the REF2021 website

The significance of my output is not evident in the text of the output itself - what do I do?

The Informatics sub-panel recognises that this might be the case, and welcomes the submission of a factual statement on the significance of the output where appropriate (max 100 words) (PCWM 256-257). See Steve's advice on writing the 100 words under Outputs in Informatics.

Outputs in Informatics

For outputs like software or standards documents, a statement of maximum 300 words is required to describe the research process and content (PCWM 254).

For reviews, the original research or new insights from the review may be reported in statement of maximum 300 words (PCWM 255).

Any and all additional statements on outputs need to be 'succinct, verifiable, and externally referenced where appropriate' (PCWM 257).

Panel criteria and working methods (REF 2019/02) from the REF2021 website

Can I update my 100 words after the REF check exercise deadline (24 April 2019)?

Yes. You will be able to edit the 100 words that accompany your nominations after the internal REF check deadline (24 April 2019). However, it is advised that you wait to receive feedback either from the 100 word workshops on the Research Day (08 May 2019) or from the internal review panel before making changes to statements submitted in time for the deadline. Once feedback is received, you can (and should) further edit your 100 words. There will be a period of College and University-level review of output scores (NB: output scores, not output content) during which 100 words cannot be edited. The exact timing of this will be communicated closer to the time, but is expected to be 28 June - end July (College) and around September 2019 (University).

How can I emphasise "rigour" in my 100 words?

You only need to address the rigour of your research if it is not evident in your paper. It is more likely that you will touch on rigour as part of describing the significance of your work, for example if your evaluation is the biggest or most thorough conducted to date.

Where can I see examples of submissions to REF2014?

The submissions and results of REF2014 are available online. For examples of good submissions to UoA 11 (Computer Science and Informatics), see University of Oxford and University College London. For examples of good supporting statements for outputs (i.e. the 100 words), see the UoA 11 submissions by University of Sheffield and University of Nottingham.

REF2014 Results & submissions website

University of Oxford UoA 11 submission to REF2014

University College London UoA 11 submission to REF2014

University of Sheffield UoA 11 output submission to REF2014

University of Nottingham UoA 11 output submission to REF2014

At what level are outputs nominated and selected for final REF submission?

Nomination, review and final selection for REF submission all happens at the unit of assessment (UoA) level. For Informatics authors, this means all the work is done at School level.

What does it mean for a submitted output to be double-weighted?

Work of extended scale and scope may be requested to be double-weighted, but the panel for Informatics does not expect many requests for double-weighting due to the pattern of publication in the field (i.e. primarily journal articles and conference proceedings). Any potential request for double-weighted outputs must be accompanied by a statement outlining how the output satisfies the criteria for double-weighting. A reserve output may be submitted should the request for double-weighting be rejected (GoS 279-283; PCWM 237-243).

If successful, a double-weighted output will count as two attributed outputs to one staff member, unless it is attributed to two members of staff (then counted as one output per one staff member) (GoS 271, 282).

Guidance on submissions (REF 2019/01) from the REF2021 website

Panel criteria and working methods (REF 2019/02) from the REF2021 website

Can I be submitted to more than one unit of assessment (i.e. with different Schools/departments)?

No. You can only be returned to one unit of assessment (UoA) (GoS 120.f). If you have a high-scoring paper which is relevant to the Informatics unit (UoA 11), then you will be returned to this unit. 

Guidance on submissions (REF 2019/01) from the REF2021 website

Do I have to consider the type of contribution I have made in the papers I nominate to REF?

Yes. You should nominate outputs that are relevant to Informatics research, and demonstrate excellence in Informatics research. If your contribution is minimal and relates e.g. only to the analysis of research in a different area, then it may not be a strong nomination for UoA 11 (Computer Science and Informatics).

Can my Informatics co-author and I both submit the same output? 

No. Regardless of the number of authors listed on an output, the output will only be counted once (GoS 268). The only exception is for double-weighted co-authored outputs (which is unlikely in Informatics, see above) (GoS 271; PCWM 226).

Guidance on submissions (REF 2019/01) from the REF2021 website

Panel criteria and working methods (REF 2019/02) from the REF2021 website

Can my non-Informatics co-author and I both submit the same output?

Yes. Provided that your co-author is part of a different submission (either from a different UoA within the University, or from the same or different UoA at different HEI) you may both submit the same output (GoS 269; PCWM 226).

Guidance on submissions (REF 2019/01) from the REF2021 website

Panel criteria and working methods (REF 2019/02) from the REF2021 website

Do I need to mark nominated co-authored papers as "co-authored" in Pure and write a statement to describe my contribution?

No. The sub-panel for UoA 11 (Computer Science and Informatics) does not require information on co-authored outputs, and will disregard any statements on any one author's specific contribution to the output (PCWM 227).

How are interdisciplinary reseach outputs evaluated by REF?

Nominated outputs can be flagged as interdisciplinary, which prompts the reviewing sub-panel to seek advise from appointed interdisciplinary research (IDR) advisers on how to best evaluate the output. This may include the output being cross-referred to another sub-panel, but cross-referral will not be automatically triggered by marking an ouput as interdisciplinary (PCWM 392). The guidance relating to UoA 11 (Computer Science and Informatics) outlines that submission of outputs which contribute to other areas alongside computer science and informatics is welcome, but suggest that outputs which applies 'routine computational methods may be better returned elsewhere' (PCWM 85). Note as above that you can only be returned to one UoA, and you will be asked to nominate your strongest research outputs as relevant to the submitting UoA.

Panel criteria and working methods (REF 2019/02) from the REF2021 website

How do I delete a paper proposed for REF in Pure?

You can no longer delete a REF-nominated paper in Pure. However, if you wish to nominate a new paper, which you believe is stronger than your previous nominations, then you can re-rank your nominations.  Any paper you wish to delete, you should move to "Alternate".  Remember to press Update and then Save.