Internal Audit Consulting Engagement: Research Reporting System (RRS)

Executive Summary
Detailed report
- Methodology and criteria
- Observations, recommendations, and management action plan
Appendix A – Criteria and conclusions
Appendix B – Survey results

Executive Summary

Introduction

The consulting engagement of the RRS was part of the Risk-Based Annual Internal Audit Plan 2012–13 approved by the Canadian Institutes of Health Research (CIHR) Governing Council (GC). This was originally conceived as an assurance engagement but was subsequently categorized as a consulting engagement on the recommendation of the Audit Committee and Chief Audit Executive.

The Canadian Institutes of Health Research

The Canadian Institutes of Health Research is the Government of Canada's agency responsible for funding health research in Canada. CIHR was created in June 2000 under the authority of the CIHR Act and reports to Parliament through the Minister of Health. CIHR's mandate is to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health-care system." CIHR comprises 13 "virtual" institutes – each headed by a Scientific Director, who is assisted by an Institute Advisory Board – which bring together all partners in the research process – the people who fund research, those who carry it out, and those who use its results – to share ideas and focus on what Canadians need: good health and the means to prevent and fight disease. Each Institute supports a broad spectrum of research in its topic areas and, in consultation with its stakeholders, sets priorities for research in those areas. CIHR funds over 14,000 researchers and trainees in universities, teaching hospitals, and other health organizations and research centres in Canada and abroad.

Research Reporting System

CIHR’s Research Reporting System (RRS) is a web-based end-of-grant research output reporting tool (an electronic final report) designed to demonstrate the use and value of funds by researchers. Hosted on ResearchNet, researchers are now required to complete their electronic final report using the RRS within a pre-determined timeframe from the authority to use funds date. The creation of an end-of-grant reporting system was recommended by both Treasury Board Secretariat and the 2006 International Review Panel report. In March 2011, RRS was activated, and the first mandatory reporting deadline for selected programs was October, 2012.

Risks addressed

This consulting engagement addresses the following potential risks identified by management as crucial to the success of the RRS: ineffective governance and accountability structures; poor quality and inaccessibility of data (including insufficient controls over integrity, availability and confidentiality); the potential burden on staff and researchers; and risks associated with overall system security. These risks are related to the TBS Management Accountability Framework (MAF) elements of Stewardship – “The departmental control regime (assets, money, people, services, etc.) is integrated and effective, and its underlying principles are clear to all staff” – Risk Management – “The executive team clearly defines the corporate context and practices for managing organizational and strategic risks proactively”– Results and Performance – “Relevant information on results (internal, service and program) is gathered and used to make departmental decisions, and public reporting is balanced, transparent, and easy to understand” – and Learning, Innovation and Change Management – “The department manages through continuous innovation and transformation, promotes organizational learning, values corporate knowledge, and learns from its performance.”

Objective

Though included in the risk-based audit plan and initially executed as an assurance engagement, the scope and objectives of the project expanded and the role of the auditor in charge became more advisory. The CAE and the Audit Committee therefore feel that the resulting project is best described as a consulting engagement, as described in section 3.4 of the Policy on Internal Audit. This allows CAE's role as the Director, Performance and Accountability, responsible for the Planning, Reporting, Measurement and Data branch, to complement and synergize with the work conducted.

The objective of the consulting engagement was to provide advice to management that the key processes (governance, data quality, researcher burden, staff burden and the IT security safeguards) that govern the implementation and use of RRS and preserve the confidentiality, integrity, and availability of information are efficient, adequate and effective. To meet this additional advisory objective, the auditor in charge attended RRS advisory group meetings and provided ongoing suggestions to support the development of the system throughout the fieldwork process.

Scope

This consulting engagement covered the operation of the RRS including the following five key issues identified by management: governance, data quality, researcher burden, staff burden and IT security.

Criteria

The criteria used for assessing the objectives were developed by internal audit staff in conjunction with the Head, Strategic Services and Manager, Planning, Reporting, Measurement and Data.

Overall opinion

The consulting engagement has concluded that the following aspects of the Research Reporting System require significant improvements.

Statement of conformance

Although not an audit, this consulting engagement was conducted in a manner that is generally consistent with the Federal Government Policy on Internal Audit and with the Internal Auditing Standards for the Government of Canada. While the internal audit function did provide advice during and after the engagement, no management responsibilities were assumed.

Disclosure of impairment to independence and objectivity

The CAE is also the Director of the Performance and Accountability Branch, a role that involves day-to-day use and management of the RRS as well as being a member of the RRS governance body. This offers a potential impairment to the independence and objectivity (IIA Standard 1100). The CAE reviewed the report for factual accuracy and as part of the quality assurance and approval process but did not request or require changes to the substance of the report's recommendations or management action plan.

Summary of internal control strengths

The following key elements of the RRS have been implemented:

a robust and extensive consultation process with both internal and external stakeholders took place regarding the electronic final report and questions contained in the RRS, validating the content and ensuring that employees are aware of its existence and reporting capabilities
based on a recommendation made while the project was ongoing, the governance of the RRS has been clarified and a working group established to address urgent issues, with clarification of roles and responsibilities ongoing
implementation and upgrading of the RRS occurred within a robust project management framework
a process exists to follow-up with researchers who have been approved for new funding and have not completed their report by their assigned deadline
a process exists to grant exemptions to the RRS, with appropriate approvals
a survey of researchers suggests that most do not find the report unduly burdensome to complete
CIHR’s Information Architect is in the process of identifying and documenting the technical and information architecture supporting the RRS, though as of the date of this report this process is not yet complete
Since the start of the project, a vulnerability assessment has occurred that has determined the system is adequately controlled and does not require a full threat and risk assessment
Since the start of the project, CIHR's performance management regime has been approved and the RRS has been assessed against the regime
Since the start of the project, the governance body has reviewed the electronic final report's open text response questions and spaces and resolved to remove several to streamline reporting and review

Summary of internal control weaknesses

The following issues regarding the RRS require management's attention:

the purpose of the RRS requires further clarification and communication to internal and external stakeholders
there is no comparable system to track results for award recipients, who represent a considerable investment of CIHR’s financial and administrative resources and a fundamental aspect of CIHR’s core mandate
the RRS lacks a streamlined, complementary and/or customizable electronic final report to efficiently track the outcomes of non-operating grants programs
the RRS has transitioned to a change management process, however a continuous improvement and communication regime has not been formalized
a data access process with accompanying guidelines regarding who can access what types of data has not been formalized
researchers are not provided with instructions or examples of high quality reporting
no automated data verification tools and/or data quality/integrity reviews are undertaken
there is no systematic process to gather feedback on the RRS from internal and external stakeholders
data definitions/limitations have not been established

Internal Audit thanks management and staff for their excellent cooperation during this consulting engagement.

David Peckham
Chief Audit Executive
Canadian Institutes of Health Research

Management agrees with the conclusions of this report.

Jane E. Aubin
Chief Scientific Officer/Vice-President, Research and Knowledge Translation

Detailed report

Methodology and criteria

The review of CIHR’s Research Reporting System was conducted in accordance with the Federal Government Policy on Internal Audit. The principal review techniques used included:

Interviews with management and staff members at CIHR;
Surveying CIHR-funded researchers who have submitted an electronic final report;
Examination of meeting minutes, system design material, policies and other relevant documents; and
Review of the RRS system, its questions and the data it collects.

Controls were assessed as adequate if they were sufficient to minimize the risks that threaten the achievement of objectives. Detailed criteria and conclusions are contained in Appendix A.

The criteria used for assessing the objective were developed by internal audit staff in conjunction with the Head, Strategic Services and Manager, Planning, Reporting, Measurement and Data.

The project was conducted in two phases between May 2012 and September 2013.

Observations, recommendations, and management action plan

The following are observations, recommendations, and management action plan to address weaknesses in the RRS.

Observation	Recommendation	Management Action Plan
1. The purpose, contents, customizability and use of the RRS requires further clarification.
The governance history of the RRS is complicated, with no clear system owner. While being developed, draft versions and modifications of the system were presented to multiple committees within CIHR for comment and approval. In addition, the management and reporting relationships of the team responsible for designing the RRS have changed multiple times during its development. The effort to make the RRS meet multiple needs within CIHR has led to a lengthy electronic final report that is not optimized to efficiently meet a single, primary need. While the nature of information collected would be useful to multiple areas within CIHR, including performance reporting, funding opportunity/research funding design, communications and knowledge translation, it does not optimally meet the requirements of a single purpose or user. This issue was identified as a priority while the project was ongoing, and clarification of the governance process has occurred, defining the system owner as the Executive Management Committee, and the RRS champion as the Chief Scientific Officer/Vice President, Research and Knowledge Translation. In addition, a Director-led working group was created during the project to address operational issues associated with the implementation of the RRS.^{Footnote 1} Risk and impact A lack of a clear purpose can produce sub-optimal questions whose responses attempt to meet many needs poorly without addressing one primary need well. The lack of a clear purpose can also increase the number of questions without balancing the burden faced by researchers, who must complete the electronic final report, against a commensurate need for data.	1a) The primary purpose of the RRS should be established and clearly defined.	1a) Responsibility Chief Scientific Officer Action Agreed. The purpose of the RRS has been documented. The CIHR intranet and internet sites will be updated to include information about RRS and additional information will be added to reminder emails that are provided to grants holders who have been asked to complete an end of grant report. Expected completion September, 2014
Review of the questions contained in the current RRS by the working group concluded that several definitions and question interpretations were obsolete or vague. Risk and impact Using obsolete, unclear or vague definitions and questions may lead to inappropriate use or divergent interpretations of the data.	1b) A coherent, updated terminology with definitions and interpretation of the questions used in the RRS should be clearly embedded and accessible within the system.	1b) Responsibility Chief Scientific Officer Action Agreed An analysis will be completed to identify a list of questions frequently asked by researchers regarding definitions and the interpretation of RRS questions, and a set of standard answers will be developed for these questions. This document will be made available to the research community through a link provided in reminder e-mails. The feasibility of making the recommended embedded system changes will be assessed. Expected completion December 2014
Throughout the data request and distribution process it is not disclosed that the electronic final report contains self-reported data by researchers that has not been independently verified or undergone a quality assurance check. Risk and impact Lack of awareness about the self-reported nature of the data risks any information contained within electronic final reports being mistaken for official CIHR figures or opinions.	1c) The limitations of the data, including its self-reported nature, problematic definitions and clarifications, should be distributed with the data whenever it is requested or downloaded from the system.	1c) Responsibility Director, Performance and Accountability Action Agreed A standard disclaimer will be developed to describe the sources of the RRS data (i.e. the self-reported nature of the data) and acceptable uses of the data. This disclaimer will be included with all requests for RRS data and in all publications that reference RRS data. Expected completion September, 2014
The RRS contains a large volume of open text that is not reviewed for problematic content and is not currently used to support any CIHR reporting or decision-making. The current version of the electronic final report can hold as many as 25,000 characters worth of open text for each electronic final report submitted. While a potentially rich source of qualitative information, open text responses are the most burdensome type of data for researchers to provide and for CIHR staff members to analyze. Comparable analysis of existing open text data at CIHR indicates it is a laborious process and no technical solution has been identified to automate it. Several groups not involved in the current review of RRS have indicated a desire for open text information but it is not clear if the need for and use of the data is commensurate with the burden of gathering, storing and analyzing it. As of the date of this report, the RRS Working Group has analyzed the appropriateness of each open-ended question and recommended a targeted reduction in the number of questions. Risk and impact Without a plan to analyze open text responses, it is purely a burden for researchers, staff and CIHR’s IT infrastructure.	1d) The open text questions should be reviewed against the uses of the RRS to determine if collecting such a large volume of text is merited and does not represent an excessive burden on researchers or CIHR staff relative to its benefits.^{Footnote 2}	1d) Responsibility Chief Scientific Officer Action Agreed. A review has been completed of the RRS question set, and recommendations have been made to develop a streamlined version which would eliminate and/or shorten a lot of the open text. A roll-out plan has been established to transition in the new reporting template. Expected completion February 2015
The RRS focuses on the results of grants (funding given to recipients to conduct research) and not awards (funding given to recipients as a salary or to complete training). There is no system comparable to the RRS to track the outputs and activities of recipients of training awards that support students as they progress towards a degree, or salary awards that support the position of an independent researcher at an institution. These are direct recipients of funds from individual funding opportunities, who may or may not go on to apply for grants or awards in the future. While the RRS does track the number of students and fellows (not independent researchers) attached to each grant as one of its “output” categories, this information is nonspecific, reported as a total with no space or authority to track individual names and degrees. This renders its use limited to track the career of specific individuals and the impact that CIHR funding has had on them. It also reflects only those trainees who are indirectly related to CIHR funding, rather than those who receive it directly. Risk and impact Training and salary awards represent a small dollar-value of CIHR’s budget overall, but directly support thousands of trainees and researchers every year. This represents a crucial part of CIHR’s mandate and strategic investment to build research capacity. Without an RRS-like system, or separate awards module to track recipient career progress and the results of funding, the process to gather data to analyze and report on these programs will be haphazard and burdensome.	1e) A method to track the training and/or career progress of award recipients should be established. Two important issues to be considered in this system are: a way of identifying unique individuals a means of encouraging recipients to complete the Electronic Final Report even if they do not plan on applying for future CIHR funding	1e) Responsibility Director, Performance and Accountability Action Agreed A method to track award recipients and trainees funded by CIHR grants will be developed. Over the short-term, a separate survey system is being used to gather this information. The feasibility of using the RRS, or the development of an alternative mechanism, will be explored, in alignment with the ongoing Reforms process. Expected completion Short-term survey: Completed Long-term system: December, 2016
The RRS is designed to capture the results of research funding for CIHR’s primary research funding program, the open operating grants. These are generally high-value, multi-year awards that aim to produce primary research and new knowledge. However, CIHR has a variety of non-open programs for which the RRS is poorly-suited, including: Dissemination awards which exist to fund a meeting or other form of research dissemination Knowledge translation programs which emphasize the creation of a novel drug, vaccine, patent or other intervention Letter of intent grants which are used to fund a larger or more complicated application for high-value programs Bridge grants used to keep promising research facilities functioning in the face of inadequate funding Strategic programs with unusual reporting requirements All are characterized by producing outputs poorly captured by the RRS or by the electronic final report being disproportionately burdensome given the intent of the program. As of June, 2013 the funding opportunity setup process requires that funding opportunities with specific strategic goals or unique expected outcomes document these distinctive factors. These programs are limited to three reporting options: Completion the full electronic final report by the recipient; Requesting an end-of-grant reporting exemption^{Footnote 3}; or Completing a modified electronic final report (see recommendation 1f) There is currently no review of the program versus the existing electronic final report or proposed modified report to identify gaps, poor alignment of the questions or definitions, and/or redundant or unnecessary questions. Risk and impact Using the baseline electronic final report to capture the outputs of unusual programs is inefficient, ineffective and frustrating for both researchers and CIHR staff members. However, simply allowing these programs to be exempt from any report means that millions of dollars’ worth of grants will not be included in CIHR’s reporting regime. In addition, these programs may address important areas of CIHR’s mandate that should be captured through some sort of customized electronic final report. In all cases the risk is that reporting is distorted and unrepresentative. Failing to specify a reporting plan in advance risks funding a program whose success cannot be measured, evaluated or assessed. Failing to specify a response in advance and relying on corporate memory risks a complete loss of the program reporting plan rationale if the individuals responsible for the creation of the program leave CIHR.	1f) A process should exist to capture the unique outputs of specific programs. The unique outputs of these programs not captured by the RRS should be documented as part of the funding opportunity set-up process to enable future reporting.	1f) Responsibility Chief Scientific Officer Action Agreed There will be a requirement for all electronic final reports to contain a standard baseline of RRS questions across all programs. The feasibility of adopting a modular approach to the RRS is currently being explored. This would allow programs to have the option of adding questions to the RRS. If the RRS is not a suitable system to capture unique program outputs, alternative options will be explored. Documentation of unique program outputs will be included as part of program design. Expected completion Feasibility analysis: March, 2015
2. The parent system that houses the RRS has not had a comprehensive security assessment since 2004.
RRS is a sub-system of the ResearchNet web portal, which is the main interface between CIHR and the research community and its uses have been growing since its inception.^{Footnote 4} Operationally ResearchNet is used to receive and process applications, manage several stages of the peer review process and communicate results of funding opportunities after approval. The RRS module will also be used to feed data into CIHR’s performance reporting infrastructure and annual reports to Parliament, meaning ResearchNet will also play a vital role in CIHR’s accountability. In 2007 a Privacy Impact Assessment recommended an updated Threat Risk Assessment (TRA) be completed for ResearchNet, but this has not yet occurred. In addition, ResearchNet was hosted off-site by PWGSC between 2008 and 2012, but migrated back to CIHR's premises in November, 2012. Its new location may introduce new risks to the system. No TRA or other comprehensive security assessment has been completed on ResearchNet since 2004 despite the substantial changes in and importance of ResearchNet for operational and reporting purposes. A vulnerability assessment (VA) is currently being conducted for ResearchNet, and is expected to be completed before the end of the 2013–14 fiscal years. Risk and impact Without a TRA to proactively identify risks to the system, the confidentiality, integrity and availability of information stored and transmitted by ResearchNet and the RRS cannot be ensured, compromising CIHR’s ability to fulfil its mandate and demonstrate results.	2a) A TRA or other comprehensive security assessment should be conducted for ResearchNet that also addresses the RRS component of the system. Any issues identified by the TRA should be formally addressed by management.	2a) Responsibility Chief Information Officer Action Agree A vulnerability assessment addressing the security risks to the system was conducted on ResearchNet (the parent system to the RRS) in March of 2014 and concluded that the risks were limited. Based on the results of the vulnerability assessment, a full TRA is not considered necessary. Expected completion Complete
3. The process to review the RRS is occurring before CIHR’s performance measurement frameworks (PMFs) have been approved.
CIHR’s PMF was submitted to Treasury Board in October, 2013 for approval. Despite a major proposed purpose of the RRS being to inform CIHR’s performance measurement infrastructure, the documents have not been completed or approved. The RRS’ questions are being reviewed to develop a new version of the electronic final report, but further changes may be necessary if new information needs are identified in the final, approved version of the PMF. Risk and impact Without defined and finalized PMFs, it is impossible to determine whether the RRS collects adequate, accurate and appropriate information to meet PMF needs.	3a) The RRS should be reviewed against CIHR’s approved PMF and updated to ensure it is adequately addressing the PMFs requirements.	3a) Responsibility Director, Performance and Accountability Action Agreed. The approach to streamlining the RRS was evaluated for alignment against the Performance Management Regime. Expected completion Complete
4. The RRS needs to transition to a change management approach.
The development of the RRS involved a lengthy consultation process with multiple internal and external stakeholders and occurred within a project management framework that brought rigor and documentation to the creation and updating of the system. The RRS was activated in March 2011 with a first round of mandatory deadlines in October 2012, collecting electronic final reports regarding the previous eight years of funding. As the RRS is now operational, a project management framework is no longer appropriate. Subsequent to its initial design and activation, significant changes have occurred regarding CIHR’s organizational structure, senior management, and funding of and reporting on grants. As new participants have been involved in the design and management of the RRS, changes are being suggested which will alter future versions of the electronic final report in significant ways and disrupt the division of individual electronic final reports into specific cohorts. Changes to questions and data structures will produce dramatically different versions of the electronic final report and data collected, and in order to compare versions, multiple versions will need to be maintained, as well as multiple qualifications made when reporting (see also observation 8b regarding the RRS data dictionary). Risk and impact Without a formal change management process with documentation of decision making, changes to the RRS may be made without documentation of the rationale.^{Footnote 5} Changes to the questions or data structure results in differences between versions of the electronic final report, years of data collection and cohorts of researchers, which impacts the efficiency of data collection, cleaning and storage, and significantly limits comparability across years and funding opportunities.	4a) The RRS should migrate from its current project management process to an equally-rigorous on-going management/governance process that ensures the decision-making rationales and corporate memory regarding the RRS are documented and preserved.	4a) Responsibility Chief Scientific Officer Action Agreed. Since its migration from an IT project to an active system, any changes to the RRS have undergone, and in the future must continue to undergo a formal system change request process which will document the rationale for changes. An RRS governance structure will be formally established as a subcommittee of the Extended Executive Management Committee (EEMC). Expected completion September, 2014
4 b) The process to change the RRS should involve the following actions to maximize comparability between reporting cohorts: involve multi-year “steady-states” in which significant changes to the data structure and questions are accumulated establish and document a process to manage differences between funding opportunity, funding year, end of grant and reporting cohorts	4b) Responsibility Director, Performance and Accountability Action A baseline dataset will be established in order to maintain comparability of RRS data over the years. Expected completion September 2014
5. There is no process to request data from the RRS and it is has not been determined who will have access to the data.
The first deadline for RRS reports was October 2012, and CIHR has accumulated nearly 2,000 electronic final reports since that date. Despite this volume of reports, requests to access the data, and in particular opportunities to use it have been limited by a lack of awareness and the lack of clarity regarding how to access it. While there is no explicit risk of not releasing the data, the existence of such a large and rich body of information is an opportunity for a large number of people to identify problems, improvements, highly significant results or potential synergies and unexpected uses of the data. It could also inform the ongoing electronic final report redesign process discussed in observations 1 and 4. There are no guidelines regarding who can access individual reports or aggregate information collected by the RRS. Access to individual reports and aggregate data by CIHR employees should be unproblematic, but there is no clarity regarding whether the Institutes, research institutions, other researchers or the general public will have access to the data, or what form that access would take. Though various policies and notifications communicated to researchers regarding the RRS can be interpreted to indicate the information may be released publicly, this possibility was not explicit. Inquiries by researchers and a systematic survey (see Appendix B) demonstrated that researchers had concerns regarding access proprietary information (e.g. certain research results involved intellectual property shared or developed by private industry) or research results that had not yet been published by the public, government or peers, and that access to such information may impact their professional lives. In addition, the significant amount of open text collected with each electronic final report is unrestricted and not reviewed, introducing the risk of embarrassing, personal or confidential information being released (see recommendation 1d). While there are justifiable concerns about the protection of personal or confidential information, the data captured by the RRS have numerous potential uses within the research community or government environment that could benefit a variety of non-CIHR stakeholders. Its public release would also provide accountability and transparency regarding the use of taxpayer funds, and would be a source of information about and access to research results for the general public. The RRS is currently storing a set of responses from researchers that have already submitted their electronic final reports, and researchers do not necessarily have an incentive or reason to revisit them. These reports would require a different strategy to offset the risk of public release (if it occurred) since researchers were not explicitly made aware of this possibility before submitting them. Risk and impact A lack of clarity regarding who has access to the RRS data hinders accessing, releasing and redistributing results, as well as the ability of researchers to tailor their replies to the appropriate audience or limit problematic information. If researchers become concerned about access to RRS results without controls regarding such information, responses may be incomplete or inaccurate. While the researchers may have consented on usage of information found in the RRS, the consent documents do not appear to clearly state “this information will be made public”. The audit-led survey of researchers indicated that such a move would be surprising to at least some researchers, and several indicated that had they known who the potential audience was, they would have given different responses. Releasing the RRS’ information publicly (if this is done) may be a point of controversy for some researchers. While CIHR may be technically correct that researches were informed their information could be released to the general public or other groups, releasing the information without explicit consent risks alienating one of CIHR’s most crucial group of external stakeholders. Failing to release the data limits the myriad possible uses of a very rich source of information.	5a) A process and guidelines should be established for requesting and releasing data from the RRS, addressing issues such as data protection, format, confidentiality and redistribution. The guidelines should discuss: Access by employees Access by nonemployees^{Footnote 6} Access to individual researcher reports Access to open text responses Restrictions governing redistribution, if needed	5b) Responsibility Director, Performance and Accountability Action Agreed The privacy review discussed in 5a) will include a review of researcher consent. The results of this review will be communicated as recommended. Expected completion December, 2015
5b) Depending on the actions taken to address recommendation 5a, an analysis of what constitutes researcher consent should be addressed and appropriately communicated.	5b) Responsibility Director, Performance and Accountability Action Agreed The privacy review discussed in 5a) will include a review of researcher consent. The results of this review will be communicated as recommended. Expected completion December, 2015
6. There is no process or guidance to assist researchers in providing accurate, high-quality data in their responses.
The data collected by RRS is self-reported by researchers, and includes several data field types (numerical, alphabetic, open and closed text). There are no automated error-checking measures for open, numerical responses, and no review of the data for accuracy, completeness or appropriateness. Performing such a review (particularly of open text responses) would also represent a significant internal burden estimated at 0.5–1 FTE per year with no guarantee of improved data quality. In addition, researchers indicated in a survey (see Appendix B) that the length of the electronic final report and technical errors in the system have resulted in them providing the fastest or easiest answers rather than the best or most accurate. This is exacerbated by the lack of training, incentives and sample responses to ensure the data meets CIHR’s needs. Risk and impact Without some sort of guidance, assistance or quality control measures, the data gathered by CIHR may be inaccurate and researchers may provide suboptimal responses.	6a) Researchers should be provided with guidance to encourage high-quality responses.^{Footnote 7}	6a) Responsibility Chief Scientific Officer Action Agreed The RRS Working Group will develop responses to questions that are frequently asked by researchers to provide instructions to researchers regarding how to provide high-quality responses to RRS questions. The Grants and Awards Guide will be updated to include the need for high-quality RRS responses in the End of Grant Reporting policy. Expected completion December 2014
6b) Where possible the system should include automated controls to ensure data quality for closed-ended numerical responses and allow the system to correct or flag problematic answers.	6b) Responsibility Chief Scientific Officer Action Agreed The change management group will review the RRS business rules, consulting with the Systems group as necessary. This group will bring forward an analysis and a plan going forward to the full RRS Working Group for discussion and approval. Expected completion April 2015
7. There is no systematic process to gather feedback from employees, researchers and other stakeholders regarding the definitions and questions contained in, and data gathered by RRS
The RRS was developed through an extensive consultation and development process involving internal and external stakeholders. Once launched, researchers completing their electronic final report were provided with an “RRS inquiries” e-mail address intended to be used to clarify any questions or issues they might have regarding the system. This e-mail address has been used as a “catch-all” to respond to all types of inquiries by researchers, including technical problems, definitions and comments regarding the system overall. Responding to inquiries has been split between technical and operational questions by two different subject matter experts. Frustrated researchers have also submitted feedback regarding the system (generally in the form of criticisms) directly to the President or the EVP Research. A systematic survey of researchers who have completed an electronic final report suggests that these responses represent the minority of researchers, most of whom did not find the system difficult to use. Risk and impact Without a formal process to collect feedback from researchers regarding the RRS, efforts to gather technical issues, problems with definitions and other comments from external stakeholders will not be completed in a systematic manner The opportunity to systematically identify and track system improvements will be lost. In addition, lacking an appropriate feedback mechanism, researchers are likely to continue their practice of appealing directly to CIHR’s senior management and the President, which risks conveying a selectively biased perspective regarding the RRS.	7a) A method of systematically gathering feedback about RRS from researchers in a timely fashion should be established. This external feedback processes should enable CIHR to identify both short-term issues with the current system and long-term issues that could feed into updates of the system in the future and should be solicited shortly after the electronic final report is submitted.^{Footnote 8}	7a) Responsibility Chief Scientific Officer Action Agreed A mechanism to systematically collect and respond to feedback from both internal and external stakeholders we be developed. Expected completion December, 2015
RRS’ development process permitted internal parties to comment on the questions and definitions used and suggest clarifications. However, since RRS “went live” and in particular since it began gathering actual data, no consultation process of comparable rigor has existed to systematically determine if data users understand the definitions used, data gathered, and its limitations. A general RRS inquiries e-mail address does exist but is not used to gather systematic feedback from employees. Risk and impact Without a systematic process to assess the understanding of the limitations of the RRS by CIHR employees, the data may be systematically misunderstood and/or misrepresented by employees.	7b) A method of systematically gathering feedback about RRS from CIHR employees regarding the system definitions, data structure and limitations should be established.^{Footnote 9}	7b) Responsibility Chief Scientific Officer Action Agreed This item will be addressed in conjunction with item 7a). Expected completion December, 2015
8. The RRS’ technology and information architecture has not been documented.
Enterprise architecture is necessary “to optimize across the enterprise the often fragmented legacy of processes (both manual and automated) into an integrated environment that is responsive to change and supportive of the delivery of the business strategy.” At the confluence of information technology and information management, information architecture is necessary to ensure information technology provides needed data accurately, in a timely and meaningful format. The risks and benefits of information architecture are not unique to the RRS, but the system is being used as a pilot project and seed system that will eventually encompass all of CIHR’s IT and IM infrastructure. Due to personnel changes and competing priorities, the RRS codebook (also known as a data dictionary) has not been finalized. A codebook is a vital part of any database, providing critical information about the meaning of data values within the database as well as metadata, default values, programming and performance information/settings, account roles and privileges, and other general information about the database and data overall. It is particularly valuable when data is complicated or when personnel turnover occurs. As of the date of this report, the RRS codebook is being reviewed as part of the overall CIHR enterprise architecture project. Risk and impact Without information architecture, the RRS risks fragmented reporting, less efficient operations, and more complicated and expensive upgrades that do not meet the needs of CIHR as a whole or the RRS’ critical stakeholders. Failure to update the codebook virtually guarantees the loss of corporate memory and limited reporting. Lacking such a meaningful record of changes over time creates a reliance on corporate memory; staff extracting data must be intimately be familiar with the data and manually update all extracts, adding to the time and expense of all data requests.	8a) CIHR’s information architect should continue documenting RRS’ technology and information architecture.	8a) Responsibility Chief Information Officer Action Agreed CIHR developed an enterprise approach to Enterprise Architecture in December of 2013. The End of Grant process, thus RRS will be addressed through this process. Expected completion March, 2015
8b) The RRS’ data dictionary should be updated and maintained as part of the information architecture project.	8b) Responsibility Chief Information Officer. Action Agree The systems (IT) group updated the data dictionary in Q4 of 2013–14 and created a process and/or schedule to maintain the data dictionary. Expected completion Complete

Observation

Recommendation

Management Action Plan

1. The purpose, contents, customizability and use of the RRS requires further clarification.

The governance history of the RRS is complicated, with no clear system owner. While being developed, draft versions and modifications of the system were presented to multiple committees within CIHR for comment and approval. In addition, the management and reporting relationships of the team responsible for designing the RRS have changed multiple times during its development. The effort to make the RRS meet multiple needs within CIHR has led to a lengthy electronic final report that is not optimized to efficiently meet a single, primary need. While the nature of information collected would be useful to multiple areas within CIHR, including performance reporting, funding opportunity/research funding design, communications and knowledge translation, it does not optimally meet the requirements of a single purpose or user.

This issue was identified as a priority while the project was ongoing, and clarification of the governance process has occurred, defining the system owner as the Executive Management Committee, and the RRS champion as the Chief Scientific Officer/Vice President, Research and Knowledge Translation. In addition, a Director-led working group was created during the project to address operational issues associated with the implementation of the RRS.^{Footnote 1}

Risk and impact

A lack of a clear purpose can produce sub-optimal questions whose responses attempt to meet many needs poorly without addressing one primary need well. The lack of a clear purpose can also increase the number of questions without balancing the burden faced by researchers, who must complete the electronic final report, against a commensurate need for data.

1a) The primary purpose of the RRS should be established and clearly defined.

1a) Responsibility
Chief Scientific Officer

Action
Agreed.

The purpose of the RRS has been documented. The CIHR intranet and internet sites will be updated to include information about RRS and additional information will be added to reminder emails that are provided to grants holders who have been asked to complete an end of grant report.

Expected completion
September, 2014

Review of the questions contained in the current RRS by the working group concluded that several definitions and question interpretations were obsolete or vague.

Risk and impact

Using obsolete, unclear or vague definitions and questions may lead to inappropriate use or divergent interpretations of the data.

1b) A coherent, updated terminology with definitions and interpretation of the questions used in the RRS should be clearly embedded and accessible within the system.

1b) Responsibility
Chief Scientific Officer

Action
Agreed

An analysis will be completed to identify a list of questions frequently asked by researchers regarding definitions and the interpretation of RRS questions, and a set of standard answers will be developed for these questions.

This document will be made available to the research community through a link provided in reminder e-mails.

The feasibility of making the recommended embedded system changes will be assessed.

Expected completion
December 2014

Throughout the data request and distribution process it is not disclosed that the electronic final report contains self-reported data by researchers that has not been independently verified or undergone a quality assurance check.

Risk and impact

Lack of awareness about the self-reported nature of the data risks any information contained within electronic final reports being mistaken for official CIHR figures or opinions.

1c) The limitations of the data, including its self-reported nature, problematic definitions and clarifications, should be distributed with the data whenever it is requested or downloaded from the system.

1c) Responsibility
Director, Performance and Accountability

Action
Agreed

A standard disclaimer will be developed to describe the sources of the RRS data (i.e. the self-reported nature of the data) and acceptable uses of the data. This disclaimer will be included with all requests for RRS data and in all publications that reference RRS data.

Expected completion
September, 2014

The RRS contains a large volume of open text that is not reviewed for problematic content and is not currently used to support any CIHR reporting or decision-making. The current version of the electronic final report can hold as many as 25,000 characters worth of open text for each electronic final report submitted. While a potentially rich source of qualitative information, open text responses are the most burdensome type of data for researchers to provide and for CIHR staff members to analyze. Comparable analysis of existing open text data at CIHR indicates it is a laborious process and no technical solution has been identified to automate it. Several groups not involved in the current review of RRS have indicated a desire for open text information but it is not clear if the need for and use of the data is commensurate with the burden of gathering, storing and analyzing it. As of the date of this report, the RRS Working Group has analyzed the appropriateness of each open-ended question and recommended a targeted reduction in the number of questions.

Risk and impact

Without a plan to analyze open text responses, it is purely a burden for researchers, staff and CIHR’s IT infrastructure.

1d) The open text questions should be reviewed against the uses of the RRS to determine if collecting such a large volume of text is merited and does not represent an excessive burden on researchers or CIHR staff relative to its benefits.^{Footnote 2}

1d) Responsibility
Chief Scientific Officer

Action
Agreed.

A review has been completed of the RRS question set, and recommendations have been made to develop a streamlined version which would eliminate and/or shorten a lot of the open text. A roll-out plan has been established to transition in the new reporting template.

Expected completion
February 2015

The RRS focuses on the results of grants (funding given to recipients to conduct research) and not awards (funding given to recipients as a salary or to complete training). There is no system comparable to the RRS to track the outputs and activities of recipients of training awards that support students as they progress towards a degree, or salary awards that support the position of an independent researcher at an institution. These are direct recipients of funds from individual funding opportunities, who may or may not go on to apply for grants or awards in the future. While the RRS does track the number of students and fellows (not independent researchers) attached to each grant as one of its “output” categories, this information is nonspecific, reported as a total with no space or authority to track individual names and degrees. This renders its use limited to track the career of specific individuals and the impact that CIHR funding has had on them. It also reflects only those trainees who are indirectly related to CIHR funding, rather than those who receive it directly.

Risk and impact

Training and salary awards represent a small dollar-value of CIHR’s budget overall, but directly support thousands of trainees and researchers every year. This represents a crucial part of CIHR’s mandate and strategic investment to build research capacity. Without an RRS-like system, or separate awards module to track recipient career progress and the results of funding, the process to gather data to analyze and report on these programs will be haphazard and burdensome.

1e) A method to track the training and/or career progress of award recipients should be established.

Two important issues to be considered in this system are:

a way of identifying unique individuals
a means of encouraging recipients to complete the Electronic Final Report even if they do not plan on applying for future CIHR funding

1e) Responsibility
Director, Performance and Accountability

Action
Agreed

A method to track award recipients and trainees funded by CIHR grants will be developed. Over the short-term, a separate survey system is being used to gather this information.

The feasibility of using the RRS, or the development of an alternative mechanism, will be explored, in alignment with the ongoing Reforms process.

Expected completion
Short-term survey: Completed
Long-term system: December, 2016

The RRS is designed to capture the results of research funding for CIHR’s primary research funding program, the open operating grants. These are generally high-value, multi-year awards that aim to produce primary research and new knowledge. However, CIHR has a variety of non-open programs for which the RRS is poorly-suited, including:

Dissemination awards which exist to fund a meeting or other form of research dissemination
Knowledge translation programs which emphasize the creation of a novel drug, vaccine, patent or other intervention
Letter of intent grants which are used to fund a larger or more complicated application for high-value programs
Bridge grants used to keep promising research facilities functioning in the face of inadequate funding
Strategic programs with unusual reporting requirements

All are characterized by producing outputs poorly captured by the RRS or by the electronic final report being disproportionately burdensome given the intent of the program. As of June, 2013 the funding opportunity setup process requires that funding opportunities with specific strategic goals or unique expected outcomes document these distinctive factors. These programs are limited to three reporting options:

Completion the full electronic final report by the recipient;
Requesting an end-of-grant reporting exemption^{Footnote 3}; or
Completing a modified electronic final report (see recommendation 1f)

There is currently no review of the program versus the existing electronic final report or proposed modified report to identify gaps, poor alignment of the questions or definitions, and/or redundant or unnecessary questions.

Risk and impact

Using the baseline electronic final report to capture the outputs of unusual programs is inefficient, ineffective and frustrating for both researchers and CIHR staff members. However, simply allowing these programs to be exempt from any report means that millions of dollars’ worth of grants will not be included in CIHR’s reporting regime. In addition, these programs may address important areas of CIHR’s mandate that should be captured through some sort of customized electronic final report. In all cases the risk is that reporting is distorted and unrepresentative.

Failing to specify a reporting plan in advance risks funding a program whose success cannot be measured, evaluated or assessed. Failing to specify a response in advance and relying on corporate memory risks a complete loss of the program reporting plan rationale if the individuals responsible for the creation of the program leave CIHR.

1f) A process should exist to capture the unique outputs of specific programs. The unique outputs of these programs not captured by the RRS should be documented as part of the funding opportunity set-up process to enable future reporting.

1f) Responsibility
Chief Scientific Officer

Action
Agreed

There will be a requirement for all electronic final reports to contain a standard baseline of RRS questions across all programs. The feasibility of adopting a modular approach to the RRS is currently being explored. This would allow programs to have the option of adding questions to the RRS. If the RRS is not a suitable system to capture unique program outputs, alternative options will be explored.

Documentation of unique program outputs will be included as part of program design.

Expected completion
Feasibility analysis: March, 2015

2. The parent system that houses the RRS has not had a comprehensive security assessment since 2004.

RRS is a sub-system of the ResearchNet web portal, which is the main interface between CIHR and the research community and its uses have been growing since its inception.^{Footnote 4} Operationally ResearchNet is used to receive and process applications, manage several stages of the peer review process and communicate results of funding opportunities after approval. The RRS module will also be used to feed data into CIHR’s performance reporting infrastructure and annual reports to Parliament, meaning ResearchNet will also play a vital role in CIHR’s accountability. In 2007 a Privacy Impact Assessment recommended an updated Threat Risk Assessment (TRA) be completed for ResearchNet, but this has not yet occurred. In addition, ResearchNet was hosted off-site by PWGSC between 2008 and 2012, but migrated back to CIHR's premises in November, 2012. Its new location may introduce new risks to the system.

No TRA or other comprehensive security assessment has been completed on ResearchNet since 2004 despite the substantial changes in and importance of ResearchNet for operational and reporting purposes. A vulnerability assessment (VA) is currently being conducted for ResearchNet, and is expected to be completed before the end of the 2013–14 fiscal years.

Risk and impact

Without a TRA to proactively identify risks to the system, the confidentiality, integrity and availability of information stored and transmitted by ResearchNet and the RRS cannot be ensured, compromising CIHR’s ability to fulfil its mandate and demonstrate results.

2a) A TRA or other comprehensive security assessment should be conducted for ResearchNet that also addresses the RRS component of the system. Any issues identified by the TRA should be formally addressed by management.

2a) Responsibility
Chief Information Officer

Action
Agree

A vulnerability assessment addressing the security risks to the system was conducted on ResearchNet (the parent system to the RRS) in March of 2014 and concluded that the risks were limited. Based on the results of the vulnerability assessment, a full TRA is not considered necessary.

Expected completion
Complete

3. The process to review the RRS is occurring before CIHR’s performance measurement frameworks (PMFs) have been approved.

CIHR’s PMF was submitted to Treasury Board in October, 2013 for approval. Despite a major proposed purpose of the RRS being to inform CIHR’s performance measurement infrastructure, the documents have not been completed or approved. The RRS’ questions are being reviewed to develop a new version of the electronic final report, but further changes may be necessary if new information needs are identified in the final, approved version of the PMF.

Risk and impact

Without defined and finalized PMFs, it is impossible to determine whether the RRS collects adequate, accurate and appropriate information to meet PMF needs.

3a) The RRS should be reviewed against CIHR’s approved PMF and updated to ensure it is adequately addressing the PMFs requirements.

3a) Responsibility
Director, Performance and Accountability

Action
Agreed.

The approach to streamlining the RRS was evaluated for alignment against the Performance Management Regime.

Expected completion
Complete

4. The RRS needs to transition to a change management approach.

The development of the RRS involved a lengthy consultation process with multiple internal and external stakeholders and occurred within a project management framework that brought rigor and documentation to the creation and updating of the system. The RRS was activated in March 2011 with a first round of mandatory deadlines in October 2012, collecting electronic final reports regarding the previous eight years of funding. As the RRS is now operational, a project management framework is no longer appropriate.

Subsequent to its initial design and activation, significant changes have occurred regarding CIHR’s organizational structure, senior management, and funding of and reporting on grants. As new participants have been involved in the design and management of the RRS, changes are being suggested which will alter future versions of the electronic final report in significant ways and disrupt the division of individual electronic final reports into specific cohorts. Changes to questions and data structures will produce dramatically different versions of the electronic final report and data collected, and in order to compare versions, multiple versions will need to be maintained, as well as multiple qualifications made when reporting (see also observation 8b regarding the RRS data dictionary).

Risk and impact

Without a formal change management process with documentation of decision making, changes to the RRS may be made without documentation of the rationale.^{Footnote 5}

Changes to the questions or data structure results in differences between versions of the electronic final report, years of data collection and cohorts of researchers, which impacts the efficiency of data collection, cleaning and storage, and significantly limits comparability across years and funding opportunities.

4a) The RRS should migrate from its current project management process to an equally-rigorous on-going management/governance process that ensures the decision-making rationales and corporate memory regarding the RRS are documented and preserved.

4a) Responsibility
Chief Scientific Officer

Action
Agreed.

Since its migration from an IT project to an active system, any changes to the RRS have undergone, and in the future must continue to undergo a formal system change request process which will document the rationale for changes.

An RRS governance structure will be formally established as a subcommittee of the Extended Executive Management Committee (EEMC).

Expected completion
September, 2014

4 b) The process to change the RRS should involve the following actions to maximize comparability between reporting cohorts:

involve multi-year “steady-states” in which significant changes to the data structure and questions are accumulated
establish and document a process to manage differences between funding opportunity, funding year, end of grant and reporting cohorts

4b) Responsibility
Director, Performance and Accountability

Action
A baseline dataset will be established in order to maintain comparability of RRS data over the years.

Expected completion
September 2014

5. There is no process to request data from the RRS and it is has not been determined who will have access to the data.

The first deadline for RRS reports was October 2012, and CIHR has accumulated nearly 2,000 electronic final reports since that date. Despite this volume of reports, requests to access the data, and in particular opportunities to use it have been limited by a lack of awareness and the lack of clarity regarding how to access it. While there is no explicit risk of not releasing the data, the existence of such a large and rich body of information is an opportunity for a large number of people to identify problems, improvements, highly significant results or potential synergies and unexpected uses of the data. It could also inform the ongoing electronic final report redesign process discussed in observations 1 and 4.

There are no guidelines regarding who can access individual reports or aggregate information collected by the RRS. Access to individual reports and aggregate data by CIHR employees should be unproblematic, but there is no clarity regarding whether the Institutes, research institutions, other researchers or the general public will have access to the data, or what form that access would take. Though various policies and notifications communicated to researchers regarding the RRS can be interpreted to indicate the information may be released publicly, this possibility was not explicit. Inquiries by researchers and a systematic survey (see Appendix B) demonstrated that researchers had concerns regarding access proprietary information (e.g. certain research results involved intellectual property shared or developed by private industry) or research results that had not yet been published by the public, government or peers, and that access to such information may impact their professional lives. In addition, the significant amount of open text collected with each electronic final report is unrestricted and not reviewed, introducing the risk of embarrassing, personal or confidential information being released (see recommendation 1d).

While there are justifiable concerns about the protection of personal or confidential information, the data captured by the RRS have numerous potential uses within the research community or government environment that could benefit a variety of non-CIHR stakeholders. Its public release would also provide accountability and transparency regarding the use of taxpayer funds, and would be a source of information about and access to research results for the general public. The RRS is currently storing a set of responses from researchers that have already submitted their electronic final reports, and researchers do not necessarily have an incentive or reason to revisit them. These reports would require a different strategy to offset the risk of public release (if it occurred) since researchers were not explicitly made aware of this possibility before submitting them.

Risk and impact

A lack of clarity regarding who has access to the RRS data hinders accessing, releasing and redistributing results, as well as the ability of researchers to tailor their replies to the appropriate audience or limit problematic information. If researchers become concerned about access to RRS results without controls regarding such information, responses may be incomplete or inaccurate.

While the researchers may have consented on usage of information found in the RRS, the consent documents do not appear to clearly state “this information will be made public”. The audit-led survey of researchers indicated that such a move would be surprising to at least some researchers, and several indicated that had they known who the potential audience was, they would have given different responses. Releasing the RRS’ information publicly (if this is done) may be a point of controversy for some researchers. While CIHR may be technically correct that researches were informed their information could be released to the general public or other groups, releasing the information without explicit consent risks alienating one of CIHR’s most crucial group of external stakeholders.

Failing to release the data limits the myriad possible uses of a very rich source of information.

5a) A process and guidelines should be established for requesting and releasing data from the RRS, addressing issues such as data protection, format, confidentiality and redistribution.

The guidelines should discuss:

Access by employees
Access by nonemployees^{Footnote 6}
Access to individual researcher reports
Access to open text responses
Restrictions governing redistribution, if needed

5b) Responsibility
Director, Performance and Accountability

Action
Agreed

The privacy review discussed in 5a) will include a review of researcher consent. The results of this review will be communicated as recommended.

Expected completion
December, 2015

5b) Depending on the actions taken to address recommendation 5a, an analysis of what constitutes researcher consent should be addressed and appropriately communicated.

5b) Responsibility
Director, Performance and Accountability

Action
Agreed

The privacy review discussed in 5a) will include a review of researcher consent. The results of this review will be communicated as recommended.

Expected completion
December, 2015

6. There is no process or guidance to assist researchers in providing accurate, high-quality data in their responses.

The data collected by RRS is self-reported by researchers, and includes several data field types (numerical, alphabetic, open and closed text). There are no automated error-checking measures for open, numerical responses, and no review of the data for accuracy, completeness or appropriateness. Performing such a review (particularly of open text responses) would also represent a significant internal burden estimated at 0.5–1 FTE per year with no guarantee of improved data quality. In addition, researchers indicated in a survey (see Appendix B) that the length of the electronic final report and technical errors in the system have resulted in them providing the fastest or easiest answers rather than the best or most accurate. This is exacerbated by the lack of training, incentives and sample responses to ensure the data meets CIHR’s needs.

Risk and impact

Without some sort of guidance, assistance or quality control measures, the data gathered by CIHR may be inaccurate and researchers may provide suboptimal responses.

6a) Researchers should be provided with guidance to encourage high-quality responses.^{Footnote 7}

6a) Responsibility
Chief Scientific Officer

Action
Agreed

The RRS Working Group will develop responses to questions that are frequently asked by researchers to provide instructions to researchers regarding how to provide high-quality responses to RRS questions. The Grants and Awards Guide will be updated to include the need for high-quality RRS responses in the End of Grant Reporting policy.

Expected completion
December 2014

6b) Where possible the system should include automated controls to ensure data quality for closed-ended numerical responses and allow the system to correct or flag problematic answers.

6b) Responsibility
Chief Scientific Officer

Action
Agreed

The change management group will review the RRS business rules, consulting with the Systems group as necessary. This group will bring forward an analysis and a plan going forward to the full RRS Working Group for discussion and approval.

Expected completion
April 2015

7. There is no systematic process to gather feedback from employees, researchers and other stakeholders regarding the definitions and questions contained in, and data gathered by RRS

The RRS was developed through an extensive consultation and development process involving internal and external stakeholders. Once launched, researchers completing their electronic final report were provided with an “RRS inquiries” e-mail address intended to be used to clarify any questions or issues they might have regarding the system. This e-mail address has been used as a “catch-all” to respond to all types of inquiries by researchers, including technical problems, definitions and comments regarding the system overall. Responding to inquiries has been split between technical and operational questions by two different subject matter experts.

Frustrated researchers have also submitted feedback regarding the system (generally in the form of criticisms) directly to the President or the EVP Research. A systematic survey of researchers who have completed an electronic final report suggests that these responses represent the minority of researchers, most of whom did not find the system difficult to use.

Risk and impact

Without a formal process to collect feedback from researchers regarding the RRS, efforts to gather technical issues, problems with definitions and other comments from external stakeholders will not be completed in a systematic manner The opportunity to systematically identify and track system improvements will be lost.

In addition, lacking an appropriate feedback mechanism, researchers are likely to continue their practice of appealing directly to CIHR’s senior management and the President, which risks conveying a selectively biased perspective regarding the RRS.

7a) A method of systematically gathering feedback about RRS from researchers in a timely fashion should be established. This external feedback processes should enable CIHR to identify both short-term issues with the current system and long-term issues that could feed into updates of the system in the future and should be solicited shortly after the electronic final report is submitted.^{Footnote 8}

7a) Responsibility
Chief Scientific Officer

Action
Agreed

A mechanism to systematically collect and respond to feedback from both internal and external stakeholders we be developed.

Expected completion
December, 2015

RRS’ development process permitted internal parties to comment on the questions and definitions used and suggest clarifications. However, since RRS “went live” and in particular since it began gathering actual data, no consultation process of comparable rigor has existed to systematically determine if data users understand the definitions used, data gathered, and its limitations. A general RRS inquiries e-mail address does exist but is not used to gather systematic feedback from employees.

Risk and impact
Without a systematic process to assess the understanding of the limitations of the RRS by CIHR employees, the data may be systematically misunderstood and/or misrepresented by employees.

7b) A method of systematically gathering feedback about RRS from CIHR employees regarding the system definitions, data structure and limitations should be established.^{Footnote 9}

7b) Responsibility
Chief Scientific Officer

Action
Agreed

This item will be addressed in conjunction with item 7a).

Expected completion
December, 2015

8. The RRS’ technology and information architecture has not been documented.

Enterprise architecture is necessary “to optimize across the enterprise the often fragmented legacy of processes (both manual and automated) into an integrated environment that is responsive to change and supportive of the delivery of the business strategy.” At the confluence of information technology and information management, information architecture is necessary to ensure information technology provides needed data accurately, in a timely and meaningful format. The risks and benefits of information architecture are not unique to the RRS, but the system is being used as a pilot project and seed system that will eventually encompass all of CIHR’s IT and IM infrastructure.

Due to personnel changes and competing priorities, the RRS codebook (also known as a data dictionary) has not been finalized. A codebook is a vital part of any database, providing critical information about the meaning of data values within the database as well as metadata, default values, programming and performance information/settings, account roles and privileges, and other general information about the database and data overall. It is particularly valuable when data is complicated or when personnel turnover occurs. As of the date of this report, the RRS codebook is being reviewed as part of the overall CIHR enterprise architecture project.

Risk and impact

Without information architecture, the RRS risks fragmented reporting, less efficient operations, and more complicated and expensive upgrades that do not meet the needs of CIHR as a whole or the RRS’ critical stakeholders.

Failure to update the codebook virtually guarantees the loss of corporate memory and limited reporting. Lacking such a meaningful record of changes over time creates a reliance on corporate memory; staff extracting data must be intimately be familiar with the data and manually update all extracts, adding to the time and expense of all data requests.

8a) CIHR’s information architect should continue documenting RRS’ technology and information architecture.

8a) Responsibility
Chief Information Officer

Action
Agreed

CIHR developed an enterprise approach to Enterprise Architecture in December of 2013. The End of Grant process, thus RRS will be addressed through this process.

Expected completion
March, 2015

8b) The RRS’ data dictionary should be updated and maintained as part of the information architecture project.

8b) Responsibility
Chief Information Officer.

Action
Agree

The systems (IT) group updated the data dictionary in Q4 of 2013–14 and created a process and/or schedule to maintain the data dictionary.

Expected completion
Complete

Appendix A – Criteria and conclusions

The project uses the following definitions to make its assessment of the internal control framework.

Conclusion on Criteria	Definition of Opinion
Well controlled	Well managed, no material weaknesses noted or only minor improvements are needed.
Moderate issues	Control weaknesses, but exposure is limited because either the likelihood or the impact of the risk is not high.
Significant improvement required	Control weaknesses either individually or cumulatively represent the possibility of serious exposure.

Overall conclusion

The consulting engagement has concluded that the Research Reporting System requires significant improvements.

Criteria	Observation	Conclusions
1. The RRS has a clear governance process and a defined decision-making authority.	Observation 1a)	Significant improvements required
2. A data access model has been defined.	Observations 1d), 5a) and 5b)	Significant improvements required
3.The RRS does not duplicate information gathered in other CIHR surveys.	Observation 1e)	Moderate issues
4. A quality control process reviews data before it is published.	Observations 1b), 1c), 6a) and 6b)	Significant improvements required
5. The RRS’ questions address reporting requirements found in CIHR’s performance measurement frameworks.	Observations 1f), 1g) and 3a)	Moderate issues
6. The RRS has a process to gather feedback regarding the questions and the system from internal stakeholders.	Observation 7b)	Moderate issues
7. The RRS has a process to gather feedback regarding the questions and the system from external stakeholders.	Observation 7a)	Moderate issues
8. The RRS has a process to modify the RRS based on feedback regarding questions and the system.	Observations 4a), 4b) and 8b)	Moderate issues
9. There is a process to review the deadlines and receipt of individual RRS submissions.	Minor recommendation, management letter	Well controlled
10. There is a process to request data from RRS.	Observation 5a)	Significant improvements required
11. Reporting requirements are documented.	Deferred to 1, 2 and 5	N/A
12. Technology and information architecture supporting reporting has been documented.	Observation 8a)	Moderate issues
13. The appropriate security reviews have taken place regarding the RRS, and recommendations produced have been addressed.	Observation 2a)	Significant improvements required

Appendix B – Survey results

Sample characteristics

As part of the project, Internal Audit surveyed 817 researchers who had completed at least one electronic final report, of which 294 submitted a complete response (response rate = 36%). The following is a brief summary of the results. Unless otherwise noted, differences between pillars were not significant.

Technical aspects

Pillar	%
Biomedical	70
Clinical	8
Health systems	8
Social, cultural, environmental and population health	12
Uncertain/don’t know	1

	The electronic final report was easy to use	The technical help I got was useful.	The content help I received was useful
Strongly agree	13%	3%	2%
Agree	52%	9%	6%
Disagree	24%	3%	4%
Strongly disagree	10%	3%	3%
Not applicable	1%	82%	85%

Time to complete the electronic final report

The modal time to complete the electronic final report was two hours, though there was a long tail of extreme outliers which brought the average time up to 9 hours.

Figure 1: Time to complete electronic final report – Overall (count) — Figure 1 long description

Hours to complete EFR # of researchers

1 16

2 69

3 39

4 29

5 29

6 15

7 1

8 16

10 22

12 8

15 12

16 3

20 7

24 5

25 3

30 4

35 3

40 8

48 1

50 2

80 1

120 1

Hours to complete EFR	# of researchers
1	16
2	69
3	39
4	29
5	29
6	15
7	1
8	16
10	22
12	8
15	12
16	3
20	7
24	5
25	3
30	4
35	3
40	8
48	1
50	2
80	1
120	1

Sharing of electronic final report

Most researchers are uninterested in the contents of other researchers’ electronic final report, and show few concerns with sharing results with other researchers, research institutions or the general public. Of the sections included in the final reports, researches showed the greatest interest in the research findings in other researchers’ electronic final report with slightly less than half expressing some interest (though many included comments that they were more likely to look in the peer reviewed literature itself). Researchers expressed the least interest in the funding information, stakeholder involvement and groups aware of other researchers’ findings.

Other final reports

Less than 10% of respondents indicated they had had to complete a similar or duplicate electronic final report for the same grant, and analysis of the results indicated that researchers were mostly discussing reports for other grants, not a duplicate report for the same grant. Based on the results of the survey, concerns regarding researcher frustration over duplicate reports do not appear to be a systematic issue.

Open text responses

227 respondents provided open text responses to at least one of three questions:

Please suggest any technical improvements or additions you would like us to consider for future releases of the electronic final report
Please provide further comments related to access to electronic final report if any
Please provide any additional comments/suggestions regarding the electronic final report (if any)

Responses were coded into categories with regard to response content rather than to which question the response was attached. All responses fell into one or more category. Four overall categories were identified (access, criticisms, report characteristics and technical problems).

Access to results

The most relevant themes regarding to access to electronic final reports were that they should be fully accessible, but require context. Respondents raised concerns regarding the ability of the audience (the general public, politicians, bureaucrats) to interpret the results and whether the electronic final reports would pre-empt publications in peer-reviewed outlets or impacting intellectual property. A small number of respondents (restricted solely to biomedical researchers) expressed concerns over their animal research being publicized. Some respondents also desired to be informed of the potential audience before they submitted their electronic final report, in part so their responses could be tailored to the audience. Some respondents also indicated their lack of interest in others' electronic final report, generally because they are redundant to the peer-reviewed literature. In addition, a minority expressed concerns that the results may be used to affect ongoing or future funding.

Criticisms of the electronic final report

A significant number of comments were critical of aspects of the electronic final report. By far the most common criticism was that the report was too long. Other areas of concern were the repetitive, redundant and unfocussed nature of the questions, a failure to understand the purpose and use of the RRS (with some calling it a purely bureaucratic or political exercise), and the poor quality of instructions provided by CIHR. Several respondents felt their electronic final report would not be read or used, and could not be interpreted. A minority of respondents reported that the electronic final report did not capture their research results.

Electronic final report characteristics

The most common comment regarding the electronic final report overall was that it required adaptation to specific grant types. A smaller number stated that specifically the electronic final report had a strong knowledge translation focus, with some respondents feeling they were disadvantaged by their focus on basic research. A small number expressed support for the existence of the electronic final report, citing accountability and public access to research results.

Technical issues

The most common technical complaint was the awkward and time-consuming method of inputting information, primarily publications. Respondents suggested using existing citation tools (i.e. digital object identifiers or pubmed identification numbers) or systems (CIHR’s Common CV) to speed data entry. Some commented that the survey used to gather their feedback came too late after completing their end of grant report.

Footnotes

Footnote 1

It should be noted that in order to meet the objectives of this project, internal audit participated in this working group and our observations are noted throughout this report.

Footnote 2

Changes to open text questions should be highlighted as part of the publication and discussion of changes to the electronic final report that will occur at EEMC and other management committees.

Footnote 3

Exemptions from the final report process involve the approval of a rational by the Associate Vice-President, Research and Knowledge Translation and the Director, Performance & Accountability. Rationales for exclusion include the nature of the activities funded (e.g. travel to a conference), an extremely low-value grant, or an existing requirement for an alternative electronic final report-type of report (in the case of jointly funded programs).

Footnote 4

Though this is a review of the RRS, as it is a component of ResearchNet its security can only be assessed in terms of the overall security of the complete ResearchNet system.

Footnote 5

While documentation of decision making may seem like a minor issue, attempts by the current governance body to discuss modifications to the current electronic final report have been hampered by a lack of background and rationale regarding the current questions and data structure.

Footnote 6

If the decision is made to make the information in RRS available to non-CIHR employees, the research community should be explicitly informed of this fact when grant funds are disbursed and as part of the reminder process once completion of the RRS is required. If the currently-stored reports are to be released, a separate risk mitigation strategy should be adopted. Options include, but are not limited to:

Preventing release of the existing reports
Review of each electronic final report by a CIHR employee
Review of each electronic final report by the researchers who submitted them, including a sign-off

Footnote 7

The feedback process discussed in 7a could be used to improve these instructions.

Footnote 8

This feedback process could also be used to review and improve the guidance to researchers discussed in 6a.

Footnote 9

Whether this internal feedback processes should be used to identify data needs or issues identified by those other than the system is deferred to the system owner.

Date modified:: 2014-11-04

Language selection

Search and menus

Search

Internal Audit Consulting Engagement: Research Reporting System (RRS)

Table of Contents

Executive Summary

Introduction

The Canadian Institutes of Health Research

Research Reporting System

Risks addressed

Objective

Scope

Criteria

Overall opinion

Statement of conformance

Disclosure of impairment to independence and objectivity

Summary of internal control strengths

Summary of internal control weaknesses

Detailed report

Methodology and criteria

Observations, recommendations, and management action plan

Appendix A – Criteria and conclusions

Overall conclusion

Appendix B – Survey results

Sample characteristics

Technical aspects

Time to complete the electronic final report

Figure 1: Time to complete electronic final report – Overall (count)

Sharing of electronic final report

Other final reports

Open text responses

Access to results

Criticisms of the electronic final report

Electronic final report characteristics

Technical issues

Footnotes

Hours to complete EFR	# of researchers
1	16
2	69
3	39
4	29
5	29
6	15
7	1
8	16
10	22
12	8
15	12
16	3
20	7
24	5
25	3
30	4
35	3
40	8
48	1
50	2
80	1
120	1

Hours to complete EFR	# of researchers
1	16
2	69
3	39
4	29
5	29
6	15
7	1
8	16
10	22
12	8
15	12
16	3
20	7
24	5
25	3
30	4
35	3
40	8
48	1
50	2
80	1
120	1

Hours to complete EFR	# of researchers
1	16
2	69
3	39
4	29
5	29
6	15
7	1
8	16
10	22
12	8
15	12
16	3
20	7
24	5
25	3
30	4
35	3
40	8
48	1
50	2
80	1
120	1