Audit of CIHR’s Commercialization Programs

Internal Audit Report

June 2014

Table of Contents


Executive Summary

Introduction

The Internal Audit of CIHR's commercialization programs is part of the 2013-14 Risk-Based Annual Internal Audit Plan which has been approved by the Canadian Institutes of Health Research's (CIHR) Governing Council (GC).

The Canadian Institutes of Health Research

The Canadian Institutes of Health Research is the Government of Canada's agency responsible for funding health research in Canada. CIHR was created in June 2000 under the authority of the CIHR Act and reports to Parliament through the Minister of Health. CIHR's mandate is to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health-care system." CIHR comprises 13 "virtual" institutes – each headed by a Scientific Director, who is assisted by an Institute Advisory Board – which bring together all partners in the research process – the people who fund research, those who carry it out, and those who use its results – to share ideas and focus on what Canadians need: good health and the means to prevent and fight disease. Each Institute supports a broad spectrum of research in its topic areas and, in consultation with its stakeholders, sets priorities for research in those areas. CIHR funds over 14,000 researchers and trainees in universities, teaching hospitals, and other health organizations and research centres in Canada and abroad.

Commercialization programs

CIHR's Commercialization and Innovation Strategy was launched in 2005, with the goal of providing a coherent framework for transforming health research into action, improve the quality of life and stimulate the economic development through discovery and innovation. The Strategy was the result of consultations with members of Governing Council, CIHR's 13 Scientific Directors, partners and stakeholders, who identified both the challenges arising from gaps in the commercialization and innovation pipeline, as well as the opportunities presented from health research

The Strategy is comprised of four primary directions – Research, Talent, Capital and Linkages. The focus is primarily on the early stages in the commercialization and innovation pipeline, where CIHR-funded efforts can directly lead to the development of new products and technologies that will improve the health of Canadians, strengthen the health system, and contribute to the growth of Canada's economy. Commercialization and innovation refer to the knowledge translation processes by which research is translated through knowledge, expertise and skilled people between the science base and its user communities contributing to the economic competitiveness, effectiveness of public services and policy, and quality of life of Canadians. This can be the component of knowledge translation that is focused on bringing new products, tools, or services to a state of use in the private, not-for-profit, or public sectors and can extend beyond simply bringing intellectual property (IP) to the marketplace for profit (e.g. cost savings in the health system, humanitarian licensing)

CIHR offered a suite of commercialization programs that encouraged the capacity of universities and teaching hospitals to interact with partners responsible for delivering the benefits of health research. It offered programs (Table 1) to support seed funding for pre-competitive research development of projects deemed to possess commercial potential, and provided opportunities for entrepreneurial training and support for technology transfer process elements and infrastructure.

As the Government of Canada's agency for health research, CIHR works with all partners in a concerted effort to move research more effectively through the marketplace to caretakers and ultimately patients that will benefit from innovations and discoveries. Through its focus on commercialization, CIHR is playing a central role in encouraging innovation that will result in solutions to the health problems that concern Canadians most as well as contribute to economic growth, investment and high-quality jobs.

All initiatives are driven by peer review, scientific excellence, high ethical standards, public and private partnerships, accountability, and, most importantly, the health of Canadians. The Institutes are integral to these activities and they, along with external partners, will continue to provide advice on and, as appropriate, co-fund existing or planned commercialization and innovation activities.

Table 1 - CIHR's commercialization programs 2011–2012Footnote 1
Program 2011-12 Actual Expenditure
Total $55,977,408
Business-Led Networks of Centres of Excellence* $1,737,000
Centres of Excellence for Commercialization and Research* $10,226,000
Grants-Networks of Centres of Excellence* $26,614,000
Small and Medium Enterprise Program (SME) $2,494,635
Collaborative Health Research Program (CHRP) $8,014,522
Proof of Principle (POP) $5,333,712
Drug Development Initiative $617,585
Science to Business (S2B) $232,500
Grants – College and Community Innovation Program $58,438
Allocated Operating Expenses $649,016
Table 1 note *

Tri-council commercialization programs where CIHR is a partner; the lead organization is another Federal granting council.

*

Risk addressed by the audit

This audit addresses the risk that CIHR's commercialization programs are not meeting their objectives and management is not monitoring these programs or modifying them if required. These risks are related to the Treasury Board Secretariat's Management Accountability Framework (MAF) elements of Policy and Programs – "Departmental research and analytic capacity is developed and sustained to assure high quality policy options, program design and advice to ministers" – Stewardship – "The departmental control regime (assets, money, people, services, etc.) is integrated and effective, and its underlying principles are clear to all staff" – and Results and Performance – "Relevant information on results is gathered and used to make departmental decisions, and public reporting is balanced, transparent, and easy to understand."

Objective

The objective of the audit is to ensure that CIHR's commercialization programs are accurately described in the program design information, have effective performance indicators that are monitored by management, and that they comply with the major components of the Treasury Board Policy on Transfer Payments.

Scope

The audit only considered commercialization programs with funds paid out in FY2011/12 where CIHR is the lead administrator, and reviewed the following programs:Footnote 2

Total $16,751,392
Small and Medium Enterprise Program* $2,494,635
Collaborative Health Research Program $8,014,522
Proof of Principle (POP) $5,333,712
Drug Development Initiative $617,585
Grants – College and Community Innovation Program $58,438Footnote 3
Science to Business $232,500
Table 2 note *

Industry Partnered Collaborative Research.

*

Overall audit opinion

The audit has concluded that there are moderate issues with CIHR's commercialization programs; however exposure is limited because either the likelihood or the impact of the risk is not high. Risks are partially mitigated as a majority of programs are in the process of sun setting and lessons learned will be used to design and structure future programs.

Statement of conformance

In my professional judgement as Chief Audit Executive, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the opinion provided in this report. The audit of commercialization was conducted in accordance with the Federal Government's Policy on Internal Audit and related professional standards. It conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of a quality assurance and improvement program.

Summary of strengths

Through the course of the audit, the following positive aspects concerning CIHR's commercialization programs were observed:

Summary of improvement opportunities

The following aspects of CIHR's commercialization programs require management's attention:

Internal Audit thanks management and staff for their assistance and cooperation throughout the audit.

David Peckham
Chief Audit Executive
Canadian Institutes of Health Research

Management agrees with the conclusions of this audit.

Jane E. Aubin
Chief Scientific Officer/Vice-President, Research and Knowledge Translation

Detailed Report

Methodology and criteria

The internal audit of CIHR's commercialization programs was conducted in accordance with the Federal Government Policy on Internal Audit. The principal audit techniques used included:

Controls were assessed as adequate if they were sufficient to minimize the risks that threaten the achievement of objectives.

Detailed criteria and conclusions are contained in the Appendix to this report.

The audit was conducted between February 2013 and April 2013.

Observations, recommendations, and management action plan

The following are audit observations, recommendations, and management action plans to address the weaknesses identified during the audit.

Observation Recommendation Management Action Plan
1. There was no consistent governance process for approving and monitoring the design of new and existing programs.

In order to understand the rationale and objectives behind CIHR's commercialization programs the audit examined both the available documentation and minutes of the meetings where decisions concerning these programs had been made.

The audit found that the details of the design of the various commercialization programs, as presented in the most recent Funding Opportunity description documents, were well developed and stated the program objectives, activities that CIHR will support as well as requirements for applications, and approaches for assessing applications.

However, when reviewing the minutes of committee meetings dating back to 2002, there was no record of a detailed review of the governance of the commercialization programs. Additionally, though staff members indicated there were post-peer review committee reviews of the individual programs' application processes, it was not documented how any committee continuously monitored these programs and made any required changes.

Risk and impact

If no formal governance processes exists surrounding the approval and monitoring of programs they will not have the opportunity to evolve and improve over time as a result of feedback and recommendations from committee members.

1a) A governance process for the approval and monitoring of CIHR's commercialization programs should be developed and documented.

1a) Responsibility

Vice President, Research and Knowledge Translation, and Director, Partnerships and Business Development

Action

Agreed.

CIHR has developed and approved a governance process for major initiatives and this process will be adopted as a best practice to guide the approval and monitoring of any commercialization programs resulting from CIHR's updated Commercialization Strategy.

This process will take into account the status of the relevant programs and overall approval and monitoring processes at CIHR.

Expected completion

April, 2015

2. The document design information and performance measurement framework for these commercialization programs were not formalized.

Throughout the audit a number of strengths were identified in the management of commercialization programs. However some of this information came exclusively from interviews with individuals who no longer worked in the commercialization programs area and could not be validated through documentation. The process to gather this information was made more complicated by the numerous reorganizations, staff turnover and file transfers within the department since the inception of many of the programs.

While all six programs examined had Funding Opportunity Description documents that clearly stated the program objectives, only two included more detailed program design documents:

  • Funding for the College and Community Innovation Program (CCIP) was requested through a Treasury Board Submission whose design documents included an analysis of risks to the program's objectives (the remaining five programs fall under CIHR's general terms and conditions for funding support which do not require a TB Submission or comparable analysis)Footnote 4
  • The Collaborative Health Research Projects Program is subject to an MOU between CIHR and NSERC that confirms the program objectives, funding arrangements, program requirements and performance measurement requirements

Except for CCIP, there was no documentation of a program risk assessment. Furthermore, no analysis was provided detailing how the program objectives are interrelated. Commercialization is included in the 2013/14 Corporate Risk Profile, but as an item within the broader "Knowledge Translation" risk, which brings limited attention to the risk that commercialization activities may not be successful.

The general requirements for performance measurement and reporting by funding recipients under the commercialization programs are well-developed and consistently applied according to Funding Opportunity detail documents. However, the performance measurement requirements in these documents do not define the specific performance measures that will be required to determine whether commercialization programs are achieving their objectives.

CIHR has established a working group to inform the development of a Commercialization Strategy, presenting an opportunity to formalize the design information and processes that should be available for all CIHR programs.

CIHR is also in the process of completing an evaluation of the commercialization programs and this evaluation framework could be used to inform future monitoring.

Risk and impact

If the program design documents do not include relevant information on objectives, risks, performance measures, and performance reporting, it is difficult to identify risks to the program's objectives, and determine if the objectives are being met. This risk is exacerbated by reliance on corporate memory and significant staff turnover. The lack of a clear performance measurement and reporting framework can also make the process to gather reporting information reactive rather than proactive, which is inefficient and may place significant demands on external stakeholders such as CIHR-funded researchers.

2a) The program design, objectives and performance measurement framework for commercialization programs should be formalized, and define any inter-relationships between the individual program the overall objective of CIHR's Commercialization Strategy.

2a) Responsibility

Vice President, Research and Knowledge Translation, Director, Partnerships and Business Development, and Director, Evaluation

Action

Agreed.

The Commercialization Strategy and/or design documents will clearly detail how any funding program supports the overall Strategy objectives and any existing inter-relations between programs.

This will take place after the Commercialization Strategy has been finalized and approved, assuming resources are made available to launch new or revised programs.

Expected completion

April, 2015

2b) The risks that could prevent the commercialization programs from achieving their objectives should be defined, documented and monitored. This process should also include the development of risk management practices to mitigate the risks and monitoring of the risks to know when mitigation should occur.

2b) Responsibility

Vice President, Research and Knowledge Translation, Director, Partnerships and Business Development, and Director, Evaluation

Action

Agreed.

Risks that could prevent the commercialization programs from achieving their objectives will be defined and documented and risk management practices will be identified. This process will take place as part of the development of the Commercialization Strategy.

Expected completion

April 2015

2c) A Performance Measurement and Reporting Framework should be developed and implemented for the commercialization programs suite

2c) Responsibility

Vice President, Research and Knowledge Translation and Director, Partnerships and Business Development

Action

Agreed.

As part of CIHR's Performance Measurement Regime (PMR), a suite of performance indicators has been identified, and will be used as the indicators available for monitoring and evaluating at CIHR. Relevant indicators will be identified from the PMR for the Commercialization Strategy and programs.

When the Commercialization Strategy has been finalized and approved, and dependant on the status of the programs, a performance measurement and reporting framework will be developed for the commercialization programs.

Expected completion

April, 2015

Appendix

Audit criteria and conclusions

The audit uses the following definitions to make its assessment of the internal control framework.

Conclusion on Audit Criteria Definition of Opinion
Well controlled Well managed, no material weaknesses noted or only minor improvements are needed.
Moderate issues Control weaknesses, but exposure is limited because either the likelihood or the impact of the risk is not high.
Significant improvements required Control weaknesses either individually or cumulatively represent the possibility of serious exposure.

The overall conclusion considers the cumulative risk exposure related to the audit observations in the context of the above criteria.

Overall conclusion

The audit has concluded that there are moderate issues related to CIHR's commercialization programs, but exposure is limited because either the likelihood or the impact of the risk is not high.

Criteria Reference to Observations Conclusion
1. Is the program design information consistent with the Treasury Board Policy on Transfer Payments? Internal Audit Report Observations #1 Moderate issues
2. Did the program design document identify what differentiates these programs from other granting programs at CIHR? Internal Audit Report Observations #2 Moderate issues
3. Did the program design information identify and address relevant risks associated with the program? Internal Audit Report Observations #2 Moderate issues
4. Was the criteria for the commercialization programs fully explained in the program design documents and is it being complied with? No exceptions noted Well controlled
5. Were performance metrics established for the commercialization programs and are they being captured? Internal Audit Report Observations #2 Moderate issues
6. Has CIHR established a monitoring regime for these programs? Internal Audit Report Observations #2 Moderate issues
7. Based on the results of the monitoring regime, is a process in place to analyse and report on the program results, and if necessary to modify programs? Internal Audit Report Observations #2 Moderate issues
8. Has CIHR established standard operating procedures to administer these programs from inception to completion? No exceptions noted Well controlled

Footnotes

Footnote 1

As per CIHR's 2011-12 Departmental Performance Report ($56.0 M)

1

Footnote 2

These programs list specific historical circumstances and several have since changed; Small and Medium Enterprise is now part of Industry-Partnered Collaborative Research, and the Drug Development Initiative was incorporated into the Proof of Principle program then retired. The initial round of funding of these programs spanned a variety of fiscal years, with the earliest (POP) being FY 2001/02.

2

Footnote 3

The College and Community Innovation Program is led by the Natural Science and Engineering Research Council, and this amount reflects the total flow-through funding administered by CIHR.

3

Footnote 4

Though a TB submission is not necessary for most CIHR granting programs, the guidance for preparing submissions does include a set of Government of Canada best practices for program design, and could inform future program design and reporting decisions.

4

Date modified: