Evaluation of the Pandemic Preparedness Strategic Research Initiative

Final Report 2014

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor Address Locator 4809A
Ottawa, Ontario K1A 0W9 Canada
Canadian Institutes of Health Research

Acknowledgements:

This report was authored by Carmen Constantinescu. The evaluation was carried out by an Evaluation Working Group which, during the different phases of the evaluation, included: Jennifer Raven, Michael Goodyer, Martin Rubenstein, David Peckham, Kwadwo Bosompra, and Danielle Halloran.

Special thanks to the all the participants in this evaluation – survey respondents, interview and case study participants, CIHR Management and Institute of Infection and Immunity (III) Management. Additional thanks to R.A. Malatest & Associates Ltd. and Science–Metrix for assisting with data collection and analysis as part of the evaluation.

Table of Contents

• Foreword
• Executive Summary
• 1. Knowledge Creation
• 2. Capacity Development
• 3. Knowledge Translation
• 4. Collaboration & Partnership
• 5. Leveraging additional funds
• 6. Program Design
• 7. Program Relevance
• References
• Appendices

Foreword

In 2006, the Government of Canada (GoC) allocated $422 million in funding to the health portfolio to support preparedness for avian and pandemic influenza. From this, the Canadian Institutes of Health Research (CIHR) received $21.5 million for pandemic influenza research, which it used to create the Pandemic Preparedness Strategic Research Initiative (PPSRI), under the leadership of the Institute of Infection and Immunity (III). Through the efforts of a Task Group formed of experts in the field of influenza and public health, four strategic research priorities for the initiative were identified: vaccines and immunization research; the virus; prevention and treatment; and ethics, legal, and social contract. Throughout these key priority areas were the themes of building research capacity, preparing for outbreak research, fostering collaborations and networks, and enhancing communication and knowledge translation.

CIHR III showed tremendous leadership in developing the PPSRI. By forming linkages with provincial, federal, and international stakeholders, III increased the total amount of available funds for research from $21.5 million to $43.3 million. As a result, 92 projects involving 345 researchers were funded in total. This represents a significant investment in, and commitment to, influenza and pandemic preparedness research in Canada.

This bibliometric evaluation indicates that the PPSRI was successful in its goal of creating knowledge. Furthermore, the outcomes produced by PPSRI-supported projects have been substantial, with 44% of survey respondents reporting new research methods, 30% reporting the development of new practices (including clinical practice, tools, and procedures), 18% of projects resulting in new or changed policies, 16% generating new vaccines or drugs, and 14% generating patents or licences. This represents a significant advancement in the field of influenza and pandemic preparedness, particularly considering that the long-term outcomes are not fully recognized.

PPSRI-investigators engaged a large number, and a wide range, of knowledge users and other stakeholders in their research projects. More than half (55%) involved healthcare practitioners, 30% involved federal or provincial government representatives, 20% engaged community or municipal organizations, and 15% included patients and healthcare consumers in their studies. This engagement of various groups demonstrates a commitment to knowledge translation and the timely and effective update of their research findings.

Although the initiative as a whole ended in 2011, many of the projects have made a lasting impact and are continuing. One key example is the PHAC-CIHR Influenza Research Network (PCIRN). This network was initially funded in 2009 to carry out research on, and inform public policy related to, seasonal influenza vaccines. Once the H1N1/09 pandemic was declared, PCIRN was swiftly tasked with evaluating the pandemic vaccines in numerous populations. In addition to renewed funding from CIHR and PHAC in 2012, PCIRN has formed several partnerships with industry groups for the testing and evaluation of influenza vaccines, clearly demonstrating the value of this network to the scientific community.

Preparing for infectious disease outbreaks remains a priority for both CIHR and III. In the 2009 Health Research Roadmap 5-year plan and in its current refresh, CIHR listed “preparing for and responding to existing and emerging threats to health” as a priority investment area. In its new 2013-18 strategic plan, III also highlighted a commitment to supporting research in this area. These pledges will ensure that Canada continues to build upon the successes achieved through the PPSRI, and assure advancement in this area.

Finally, I would like to thank Dr. Bhagirath Singh and Carol Richardson, who initiated work on the PPSRI, for their leadership. Since the transition of III to Quebec City, PPSRI has been under the competent leadership of Dr. Jennifer Raven. The success of the initiative is dependent on the positive collaborations with many stakeholders and I would like to highlight the positive interactions with Dr. John Spika; Director General of the Centre for Immunization and Respiratory Infectious Diseases at PHAC. This report is the result of excellent collaborative work between Jennifer Raven at III and Carmen Constantinescu and Martin Rubinstein from the CIHR Resources Planning and Management portfolio. Their efforts and dedication are much appreciated and this report is an important step to demonstrate the impact and capture the outcomes of the contributions of Canadian researchers in the field of pandemic preparedness.

Marc Ouellette PhD
Scientific Director,
Institute of Infection and Immunity, CIHR

Executive Summary

Pandemic Preparedness Strategic Research Initiative (PPSRI)

Influenza is an infectious disease that has caused millions of deaths in the past, and which continues to cause the death of several thousand people worldwide each year. In response to the pandemic threat posed by the emergence of a highly pathogenic strain of avian influenza (H5N1) capable of transmission from birds to humans, in May 2006 the Government of Canada committed a total of $1 billion over 5 years to improve Canada’s ability to respond to pandemics, involving the following organizations: Public Health Agency of Canada (PHAC), the Canadian Food Inspection Agency (CFIA), Health Canada (HC) and the Canadian Institutes of Health Research (CIHR).

As the Canadian funding agency for health research, CIHR received $21.5 million over 5 years (2006-2011) to support pandemic influenza research.¹ The Institute of Infection and Immunity (CIHR-III) created the Pandemic Preparedness Strategic Research Initiative (PPSRI) to develop a coordinated and focused pandemic preparedness research program, and to build national research capacity in this area. By forming partnerships with other federal and provincial agencies, CIHR-III increased the total funds available to $43.3 million. Ninety-two research projects in total were funded to address the strategic priority areas of the initiative.²

Evaluation purpose and scope

The purpose of this evaluation was to assess the relevance and performance of the PPRSI to inform decision-making regarding future similar initiatives by CIHR and to address the Treasury Board Secretariat (TBS) requirements for the specific core evaluation issues as described outlined in the TBS evaluation policy suite.³

The evaluation covers the entire period of the PPSRI (2006-2011) and builds on, and adds to, the findings of a mid-term evaluation conducted in 2009. In particular, the evaluation explored the extent to which PPSRI has met its objectives and highlights lessons learned in relation to program design and implementation. PPSRI’s success was assessed in terms of: knowledge creation, capacity building, knowledge translation, collaboration and partnership as well as leveraging additional funds.⁴

Key Findings

Knowledge creation

PPSRI funded projects were successful in generating new knowledge in the field of pandemic and influenza research. As compared to the ten most productive countries in the field of pandemic research for the period from 2001-2010, Canada has increased its specialization, as well as its scientific impact, in this area, since 2006 when the PPSRI began. Also, the scientific impact per paper based on Average of Relative Citations (ARC) for CIHR researchers funded through the PPSRI is significantly greater (1.30) than that of the rest of Canadian health researchers publishing in this area (0.68).

Across the world, the number of pandemic research papers has increased over the years for the period 2001-2010, and PPSRI funding contributed to Canada maintaining its middle rank within the ten most productive countries in the area of pandemics research. Over the period from 2008 to 2010, the percentage of pandemic research papers supported through PPSRI funding relative to the total of Canadian pandemic research papers increased from 27% to 62% with a steep increase in the number of pandemic papers produced in Canada from 37 to 103.

PPSRI-funded researchers produced a variety of publication types and attracted significant attention from the media. On average each PPSRI funded project produced 3.6 refereed journal articles; 6.9 conferences, symposium presentations and posters; 3.0 newspaper articles; 3.4 radio reports and interviews; and 2.3 internet articles. As some of the funded PPSRI projects are ongoing or have just finished, it is expected that more publications will be produced in the coming years.

Capacity development

PPSRI contributed to building capacity in Canada in the area of pandemic research by increasing the number of research staff involved in this field, this way ensuring that a pool of researchers will be able to respond to future pandemic threats. Each funded project involved an average of 11.2 research staff and it is estimated that a total of 1043 research staff were involved through all the funded projects⁵.

This evaluation demonstrated increases across all measures of training and mentorship since the mid-term evaluation. Nearly all (90%) NPIs reported that their projects included mentoring of students in the field of pandemics research. The mean number of students/trainees per project who completed their qualifications during the tenure of their mentor’s grant was 2.5. The projected total of students completing their qualifications was 224, compared to 173 in the mid-term evaluation.

A substantial proportion of both NPIs (83%) and COPI/OPIs (58%) considered their PPSRI project to have made a positive contribution to their career. The initiative focused the work of 68% and reoriented the careers of 58% of NPIs on influenza and pandemic preparedness research.

Knowledge translation

PPSRI funded projects produced a variety of research and commercialization outcomes in each research area of interest. Over a third of projects researching vaccines and immunization produced results that could lead to a new vaccine or drug. A couple of examples are: early detection of influenza variants and their correlation with variation in vaccine effectiveness, studies which identified that current vaccines are not as effective in the elderly, leading to the development of better vaccines for this age cohort. Three-quarters of the projects researching virus biology and diagnostics resulted in new research methods, patents and licenses, or intellectual property claims.

At the time of this evaluation, an increasing number of researchers funded under the PPSRI (88%), reported that they had worked on projects which included stakeholders, compared to the mid-term evaluation (66%). The most frequently reported stakeholders involved were: health system/care practitioners, study stakeholders (who were formally listed in the grant application), federal/provincial representatives and health system/care professional organizations. Generally, stakeholders were most commonly involved in data collection and the implementation of projects.

Examples of PPSRI-funded research impacts include: significant contributions to the world’s pandemic response systems by better informing disaster and humanitarian response programs of organizations such as: the Bill and Melinda Gates Foundation (BMGF), Public Health Agency of Canada (PHAC), World Health Organization (WHO) and the Ontario Health Plan for an Influenza Pandemic; research findings being adopted by the Colleges of Physicians and Surgeons in Ontario and incorporated into Nova Scotia’s policies for health care workers; PPSRI research findings stemming from mathematical modeling in infectious disease and influenza being used nationally and internationally to predict the rate of spread of H1N1.

Collaborations, partnerships and leveraging of additional funds

CIHR-III was effective in generating early collaborations with other stakeholder organizations and establishing funded partnerships for this initiative, attracting 44% of PPSRI’s financial resources from external partners. The Institute was effective in communicating with researchers by providing meetings and workshops to promote the initiative and help them with the application process.

Almost all NPIs (85%) and COPI/OPIs (82%) reported that their PPSRI grant had helped them or their teams to collaborate with other researchers involved in influenza and pandemic preparedness research. The proportion of papers published by PPSRI-funded researchers in collaboration with other researchers is higher than Canadian and world averages for pandemic research at both national and international levels.

At the project level, NPIs (45%) and COPI/OPIs (24%) were successful in leveraging additional funds as a result of their PPSRI grants. Survey respondents reported leveraging almost $25 million in total; approximately $8 million of that from additional CIHR grants. Apart from CIHR, the most frequent source of additional funding was provincial government organizations ($3 million) but the largest amount of external funding came from industry ($6.5 million) and was due primarily to a single partnership focused on vaccine evaluation.

Program design

PPSRI had several strengths including: its innovative format to rapidly respond to the need of improving pandemic preparedness and response in Canada; and, the availability of flexible funding (e.g., catalyst grants, team grants, operating grants, development and planning grants) which allowed researchers to shift focus if required. A total of 31 competitions were launched (including some priority announcements (PAs) through CIHR Open Operating Grants Program OOGP), which resulted in 92 funded grants that addressed the identified research priorities. Researchers were positive about the PPSRI grant application and decision process and many commented that the competition process was effective and administered well by CIHR.

A few suggestions for improvement were made. Partners commented on the need to improve the dissemination of the information offered to decision makers by providing it in a format that is more appropriate for them, such as synthesized summaries of key findings by theme.

Most NPIs comments related to funding - the need for additional funding, funding renewal, or funding for longer time periods, given that the PPSRI grants durations varied on average between one to maximum of three years. These findings support the earlier suggestions made in the mid-term evaluation where key informants also noted the length of the grants as a challenge and suggested that this type of initiative to include a two-year planning and implementation phase followed by five years of funding.

Another suggestion coming from researchers was to promote the creation of a consortium of vaccine manufacturers that would be capable of rapidly collaborating with researchers to create new vaccines or other therapeutics during a pandemic.

Program relevance

PPSRI’s mission was clearly aligned both with CIHR’s mandate and with the Government of Canada’s priority to fund pandemic preparedness research and all interviewed key informants considered it to be the role of the federal government to support influenza and pandemic research.

PPSRI provided additional funding on top of CIHR’s existing programs, thus supporting projects that would not otherwise have been funded in the needed area of pandemic research.

Researcher capacity, networks and collaborations have increased as a result of the PPSRI. While it is not clear how many of these gains will be sustained when funding ends, there is nevertheless a larger pool of trained people who can be called on in the event of another pandemic. NPIs emphasized the need to continue funding in pandemic preparedness research as the threat of pandemics remains given the biology of viruses.

Conclusions

The PPSRI initiative was successful in generating new knowledge, increasing research capacity and in building networks of researchers with expertise in pandemic and influenza research. Research was supported that most likely would not have otherwise been supported through CIHR’s usual funding mechanisms due to a lack of appropriate funding tools. The occurrence of the H1N1 pandemic midway through the initiative, and the fact that the PPSRI aided in Canada’s rapid response to the pandemic, demonstrated the ability of such an initiative to successfully support a national response to a public health emergency.

Qualified, multidisciplinary research teams were supported through a variety of grant types (e.g., catalyst grants, team grants, operating grants, development and planning grants) providing more flexibility to researchers. Research teams and networks were able to rapidly receive funding and respond to the pandemic threat. The process supported a breadth of projects, methodologically ranging from curiosity-driven laboratory research to large networks and team projects.

PPSRI was effective in facilitating the development of collaborations and partnerships between researchers, health service providers, and policymakers. These collaborations enabled findings from PPSRI funded research projects to have a direct impact on policy development and service delivery during the H1N1 pandemic. CIHR-III established funding partnerships with other organizations including other government departments, industry and NGOs; however, improvement could be made by better involving partners in the dissemination of information decision makers.

The most frequently identified challenge was related to the potential lack of sustainability of the infrastructure and networks established during the PPSRI and the short duration of the grants. Nevertheless, PPSRI contributed to building capacity in Canada in the area of pandemic research by increasing the number of research staff in this field and by making positive contributions to their careers, this way ensuring that a pool of researchers is positioned to rapidly respond to a future pandemic threat. With a view towards sustainability, it would be useful to clearly articulate to all stakeholders, at the launch of the strategic initiative, the funding options that would be available upon the sunset of the initiative.

Lessons learned

Findings from the evaluation identified a number of lessons learned from the PPSRI that have the potential to inform future CIHR funding initiatives. Overall, the evaluation shows that a targeted funding mechanism such as the PPSRI is an effective way to create knowledge and increase research capacity in a specific area. In particular, the following key lessons learned were identified:

• Early networking and establishing of partnerships is a key factor of success for initiative development and leveraging additional resources.
• Continuous, focused communication with all stakeholders is critical to address their information needs:
  • with researchers, it is important to provide workshops, information sessions on application development, as well as inform them about outcomes, impacts and lessons learned.
  • with partners and research users, it is important to keep them informed about the outcomes, impacts and lessons learned, as well as to understand and address their needs for information dissemination.
• Offering a variety of funding opportunities allows more flexibility for researchers to respond (given their interests, expertise and capacity the type of grants that interest them) and partners to collaborate (given their interests, expertise, resources and timelines) thereby increasing the chances of funding a rapid response to address the identified need.
• The length of the PPSRI grants, which varied between one and maximum of three years, was identified as a challenge by funded researchers. A potential means of mitigating this challenge would be to allow for a two-year planning and implementation phase prior to the five years of funding should be considered in launching a strategic initiative.

Recommendations

The five-year term of the initiative has ended, yet, the following recommendations are being made for the benefit of similar programs that may be introduced in the future and also to address the continued need for pandemic preparedness research identified by the federal government, researchers, partners and other stakeholders.

1. PPSRI has helped position Canada within the top five countries globally in the field of pandemic research and would be useful to maintain or improve upon this status. Given the ongoing threat of a future pandemic and the importance of having the capacity to respond to this threat, regular environmental scanning and assessments with international benchmarks should be conducted to determine the relative global position of Canada in the field of pandemic research in the absence of the PPSRI. This would include examining the Canadian investment in the field of pandemic research, tracking the subsequent careers of PPSRI principal investigators and trainees, and monitoring the application and use of the research on partners and research users to better understand the initiative’s influence on the field of pandemic research.
2. CIHR-III was effective at establishing funded partnerships for PPSRI, attracting almost half of the initiative’s financial resources from external partners; however, partners suggested that improvements could be made for the dissemination of information to decision makers by providing it in a format that is more appropriate for them, such as synthesized summaries of key findings by theme. In order to ensure the effective dissemination of information, future initiatives of this type should:
   1. engage the partners/knowledge users in designing of the new initiatives;
   2. include an integrated knowledge translation (iKT) plan to engage and involve partners and knowledge users in order to understand and address their needs related to the research to be undertaken and for dissemination and application of research results.

1. Knowledge Creation

One of the main objectives of PPSRI was to support the development of pandemic research in Canada. This initiative was intended to support research in areas including: developing diagnostics, antiviral drugs and vaccines; understanding and preventing disease transmission; and identifying ethical ways to use limited resources during a pandemic. It was expected that the outcomes of the research would lead to increased knowledge, and ultimately contributing to the body of knowledge used to develop the national pandemic preparedness plan.⁶

In addition to traditional methods of data collection such as online surveys and interviews with funded researchers, a bibliometric analysis was employed to assess the knowledge created in relation to the funding offered through PPSRI. Bibliometrics seeks to measure, among other things, the volume of a researcher’s publications and the relative frequency with which they are cited as a proxy for an article’s impact. The assumption underlying the use of bibliometric indices as measures of knowledge creation is that academic papers published in widely circulated journals facilitate access to the latest scientific discoveries and advances, and are seen as one of the most tangible outcomes of academic research (Goudin, 2005; Larivière et al., 2006; Moed, 2005).

For the purpose of this study, bibliometric analysis measured the publications’ scientific impact as well as the volume of publications produced, by searching Thomson Reuters’ Web of Science (WoS) for all papers having the words “influenza” and “pandemic” in their titles as well as all the papers published in the journal Influenza and other Respiratory Viruses, for all researchers funded through PPSRI (N=345 including PIs=61, and COPIs/OPIs =284).⁷ The results were compared with similar data for the top ten most productive countries in the area of pandemic research.

Scientific Impact

An important indicator of scientific performance is the impact that the published papers have on the scientific community. The Average of Relative Citations (ARC) is an indicator of the number of times that a paper is cited relative to the world average. Specialization Index (SI) is an indicator of the relative research intensity of publication of a country in an area of interest relative to the intensity of the world in that specific area. Looking at these two indicators gives us a graphical representation of the relation between knowledge creation and scientific impact. The scatterplot diagrams presented in Figure 1-1 compares the ARC and SI of the top ten productive countries in pandemic research for the period before PPSRI was implemented (2001-2005) with the period after its implementation (2006-2010). The comparison of these two figures shows that Canada has increased its specialization as well as its impact since the PPSRI started. From 2001-2005 to 2006-2010, Canada’s SI in the area of pandemic increased from 1.18 to 1.31 and its ARC, from 0.81 to 1.16.

Figure 1-1: ARC and SI of the top ten productive countries in pandemic research, 2001-2010

Figure 1-1 long description

Analyzing the scientific performance within Canada, the scientific impact per paper, based on ARC for CIHR researchers funded through PPSRI, is greater⁸ than that of the rest of Canadian health researchers publishing in the pandemic area. PPSRI-funded researchers published papers with an ARC of 1.30, as compared to 0.68 for Canadian researchers who did not received PPSRI funding, and who published in the field of pandemic preparedness.

Volume of publications

Volume of publications, although not an indicator of quality of the papers, reflects research capacity by showing the volume of papers being produced by researchers in a given research area. It is used generally as a basic indicator for the outputs that result from investing in a given program.

For the period from 2001 to 2010, Canada was situated in the “middle” of the ten most productive countries in the field of pandemic research with an average of 5.9% of the world’s pandemic papers; the United States led with the highest average percentage of pandemic papers published (Table 1-1). Over this period, the number of pandemic papers increased for the top five most productive countries (including Canada), and PPSRI contributed to Canada keeping pace with the United Kingdom, Japan and China (Figure 1-2).

Table 1-1: Ranking of the top ten most productive countries, based on the average percentages of the world pandemic papers for the period 2001-2010

Figure 1-2: Number of pandemic papers of top five most productive countries, for the period 2001-2010

Figure 1-2 long description

Over the ten year period from 2001 to 2010, the number of Canadian pandemic papers increased from a low of 13 in 2001 to a high of 103 in 2010, with a steep increase in the last three years when the PPSRI funding was available from 37 papers in 2008 to 103 papers in 2010 (Figure 1-3). The percentage with which researchers funded through the PPSRI contributed to the overall number of Canadian pandemic papers ranged from a low of 42% in 2005 to a high of 73% in 2008; however, there was a gradual increase in the proportion of papers authored by recipients of PPSRI funding with PPSRI-supported papers representing 27% of the Canadian publications in the area of pandemics in 2008, to 55% in 2009 and 62% in 2010 (Figure 1.3).

Figure1-3: Percentage of Canadian pandemic papers authored by CIHR-PPSRI-funded researchers, 2001-2010⁹

Figure 1-3 long description

Types of research outputs produced

The online survey of PPSRI-funded researchers also reveals data about researchers’ publication outputs. Conferences and publications were the most frequently reported outputs for PPSRI-funded grants (96% and 78% of projects, respectively) (Table 1-2). Overall, at the time of the survey, a total of 180 refereed journal articles have been produced, with a mean of 3.6 per project, and a total projected number of 325 refereed journal articles for all funded projects.

When comparing the overall means, the statistically significant highest rates of refereed journal articles were reported for projects with higher-value grants (greater than $500,000) (median = 9, mean = 8.8)¹⁰.

PPSRI used different types of grants (also called “funding mechanisms”) to support research in the pandemic area. These include: catalyst grants, operating grants, team grants, the Influenza Research Network (IRN) and other planning and development grants. Each of these grants had different characteristics in terms of the objectives, duration and the amounts awarded¹¹. To account for the differences in dollars expended and grant duration between the different types of grants, the average number of journal articles published per grant was normalized by dividing the total number of articles per grant by expenditures, grant duration and the number of researchers involved at the time of application, for three types of grants: catalyst, operating and team grants; IRN and the other planning and development grants, although reported publications, were excluded from this analysis due to their specific characteristics (i.e. grant size and duration), which makes for an unsuitable comparison. (Table 1-3).

The results show that there is no statistically significant difference between these three types of grants for all three control variables.¹² Although the researchers funded under the team grants produced more refereed journal articles, when we control for dollars expended, grant duration and number of researchers involved at the time of application, the differences disappear. The small number of team grants (n=5) should also be considered when interpreting these results. However, the same kind of results were reported in the evaluation of CIHR’s Open Operating Grants Program (OOGP); the publication productivity for most researchers was very similar across all CIHR pillars when grant duration was controlled for (Source: Evaluation of OOGP, 2012).

The availability of similar publication data for CIHR’s Regenerative Medicine and Nanomedicine Initiative (RMNI)¹³ and OOGP allows for a comparison of average number of refereed journal articles with PPSRI. Overall, PPSRI funded grants generated an average of 1.6 journal articles per $100K, compared with 1.5 for RMNI and 2.6 for OOGP. When normalized by grant duration, PPSRI funded grants generated an average of 2.5 journal articles per year of grant, comparing with 2.8 for RMNI and 2.3 for OOGP. These comparisons should be treated with caution due to important differences between the programs in terms of: program duration (e.g., RMNI funding period was from 2004 to 2010 compared to the PPSRI funding from 2006 to 2009); diversity of funding mechanisms used (e.g., PPSRI had the greatest variety of grant types: catalyst, operating, team, planning and dissemination); and different areas of research supported (e.g., respectively pandemics, regenerative medicine and nanomedicine, and open research).

In interviews with funded researchers, a common theme was that outputs for PPSRI projects were still being produced.¹⁴ Accordingly, the full impact of this program will not be known for some time. For example, one funded researcher reported that their project had so far resulted in 24 journal articles, with an additional 10 to 15 articles still being prepared. The evaluation of the CIHR OOGP program, which is the largest CIHR funding program, found that the number of publications peak three years after the competition year and can continue for up to nine years after the competition year. The same report shows that the average time to publish the first journal article from the start of the grant was 2.18 years (OOGP Evaluation Report, 2012). Assuming similar publication behavior between PPSRI and OOGP funded researchers, it is likely still too early to fully capture all the publication outputs generated by PPSRI funding, with the reported publications in the survey an underestimation of the total PPSRI publications.

Researchers and partners also reported attracting significant attention from international media and health organizations even though they are still in the publishing phase. The H1N1 pandemic focused media attention on the research in a way that may not have otherwise occurred. For example, one of the interviewed NPI’s was a media spokesperson for the pandemic. Another reported receiving “effective lay communication through TV, radio and print media” about a training module which aimed to prepare hospital workers to face the stresses of the pandemic.

2. Capacity Development

One of the overarching objectives identified for PPSRI was the need to build research capacity in Canada, so that in the event of a pandemic outbreak there would be sufficient research expertise necessary to address the issue. Some of the funding opportunities had as part of the eligibility requirements that researchers must include trainees on their grants.¹⁵

In assessing capacity development the evaluation looked at the number of research staff involved in the funded projects, number of trainees receiving formal qualifications on PPSRI projects and researchers’ perceptions about the influence that PPSRI had on their careers.

Number of Research Staff Working on PPSRI-Funded Projects

According to the CIHR Electronic Information System (EIS) data base, PPSRI involved a total of 345 investigators: 61 NPIs and 284 COPIs and OPIs were registered on the funding applications.

The online survey provided more information about the number and variety of the research staff working on PPSRI projects (NPIs reported on up to two projects).¹⁶. A total of 573 research staff worked on the 51 projects surveyed: an estimated 1,043 across all 92 PPSRI projects (Table 2-1)

The median number of research staff per project was 5.5, although teams varied from solo investigators to large networks (mean of 11.2). The larger research teams were generally associated with those projects funded under team or catalyst grants, the Influenza Research Network,¹⁷ and those with CIHR commitment values of greater than $250,000.

Clinical projects, which included the Influenza Research Network, generally had greater numbers of post health professional degree holders, and Master’s students, compared to the projects funded under the other CIHR pillars. Projects funded under health systems and services tended to involve more research assistants and technicians.¹⁸

Data from the bibliometric study show an increase in the number of Canadian authors publishing in the area of pandemics for the period 2001-2010, with the steepest increase occurring after 2008, from 106 Canadian authors in 2008 to 407 in 2010 (Figure 2-1).¹⁹ The onset of the H1N1 pandemic should be also considered as a factor contributing to this increase.

Figure 2-1: Number of distinct Canadian authors publishing in the area of pandemics, 2001-2010

Figure 2-1 long description

Students Trained

Nearly all (91%) of NPIs reported that their projects included mentoring of- students in influenza and pandemic preparedness research. Most NPIs reported that the projects provided interdisciplinary training to students (80%, to some or a great extent) and increased the number of trainees in influenza and pandemic preparedness research (84%). Student mentorship was provided by partners as well as by NPIs and their teams. One PhD student interviewed for a case study compared his/her involvement in the program to an apprenticeship and indicated that the experience provided direction to his/her doctoral research. A funded researcher reported a post-doctoral fellow achieved their qualifications midway through their project and attained a faculty position. In their new position, the student-turned-faculty member continued to collaborate on the project and brought a student from their own lab onto the project.

The mid-term evaluation report concluded that, although there was no baseline for comparison, it appeared that capacity building was occurring through training and mentorship on PPSRI projects.²⁰ This evaluation result demonstrates an increase across the same measures of training and mentorship (Table 2-2).

At the time of the survey, 124 students had completed their formal qualifications while working on the 51 projects reported by NPIs. The average number of students/trainees per project who achieved qualifications was 2.5 (Table 2-3), similar to the data reported for the mid-term evaluation. However, the projected total of students achieving qualifications was 224, compared to 173 at the mid-term evaluation.²¹

Relatively more postdoctoral fellows and Master’s students than doctoral students completed their formal qualifications while working on projects with a high CIHR commitment-value grant.²²

Achievement of qualifications is not the only measure of student training for PPSRI-funded projects. One of the case study projects developed an internship program, where postdoctoral students worked for three to six months at internships in Vancouver, Toronto or Montreal. The internships provided students with an opportunity to receive training and mentorship from PPSRI NPIs. The internships provided students with access to “real-life” data from the H1N1 pandemic, excellent supervision, and networking opportunities.

PPSRI Contribution to Researchers’ Careers

A substantial proportion of both NPIs (83%) and COPI/OPIs (58%) considered their PPSRI project to have made a positive contribution to their career (Figure 2-2). PPSRI re-oriented the careers of 58% and focused the work of 68% of NPIs on influenza and pandemic preparedness research. This was the case for fewer COPI/OPIs, possibly because many were earlier in their careers or were specialists in associated fields such as public health. Researchers (NPIs) who felt that the PPSRI contributed to a re-orientation of their career increased to 58% from 30% in the mid-term evaluation.²³

Figure 2-2: Contribution of PPSRI to Researchers’ Careers

Figure 2-2 long description

Source: NPI online survey QF3 and COPI/OPI online survey QC1; percentage of researchers who reported “yes”.

In interviews, researchers explained that the PPSRI grants provided them with opportunities to broaden the scope of their research, helped their career progression, and helped recruit students who had not previously worked on influenza and pandemic research.

One researcher reported that the program allowed them to undertake high-risk activities and study topics previously considered “controversial” in their field. Another researcher, involved in vaccine research, reported that the PPSRI grants gave them greater independence from industry which enhanced their credibility in the public health universe.

Also during the interviews, many researchers noted that while PPSRI funding provided them with the opportunity to take on and train students, they faced a challenge to retain those students after the end of PPSRI funding. One partner noted that graduate students had developed areas of expertise while working on PPSRI projects and the demand for these skills declined at the end of funding.

3. Knowledge Translation

At CIHR, knowledge translation (KT) is defined as a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products and strengthen the health care system. This involves a complex system of interactions between researchers and knowledge users which may vary in intensity, complexity and level of engagement depending on the nature of the research and the findings as well as the needs of the particular knowledge user.²⁴

KT was a central component of the PPSRI. Funded researchers were expected to collaborate with research users, such as public health practitioners, policy makers, industry. This was to ensure that the research findings and outcomes of funded projects were applied in concrete practices and procedures, new or updated policies, commercial products like vaccines and other therapeutics, patents or licenses, and new spin-off companies.²⁵ KT is critically important because while discoveries and generating new knowledge have the potential to result in improvements to health and health systems, these benefits will not be realized unless knowledge is put into action (Graham & Tetroe, 2007).

In assessing PPSRI’s success in knowledge translation, this evaluation looked at the type of KT and research outcomes produced, the extent of stakeholders’ involvement, and the influence that PPSRI-funded research had on these stakeholders and research users.

KT and Research Outcomes Produced

The survey of PPSRI-funded researchers reveals a large variety of KT and research outcomes reported by these researchers for their projects (Table 3-1). The most commonly reported outcomes, as reported by NPIs, were new research methods (44%), new practices or theories (30%), and changed policies or programs (18%).

Data available from the recent evaluation of RMNI and OOGP programs (Source: RMNI evaluation, 2013) allow benchmarking; however these data should be interpreted carefully due to the average grant size and the funding period of these programs: RMNI invested $80M to support research in the fields of regenerative medicine and nanomedicine, and held funding competitions between 2004-2010, and the OOGP is CIHR’s biggest open funding program which has been running since 2000.

Different research areas and pillars produced different types of results. Over a third of projects researching vaccines and immunization produced results that could lead to a new vaccine or drug. Nearly half of these projects also resulted in new practices that could include tools, instruments, procedures, and techniques. Three-quarters of projects researching virus biology and diagnostics produced new research methods, 42% produced patents and licenses, and 33% produced intellectual property claims.²⁶ Slightly over two-thirds of projects focused on prevention and treatment produced new theories. Projects focused on ethical, legal or social aspects of influenza research and pandemic preparedness were most likely to replicate and adapt research findings; influence the development of new, or change pre-existing, policies and programs; and result in new practices.²⁷ Only biomedical projects reported creating new vaccines/drugs and new products/licenses, and made intellectual property claims. Projects falling within CIHR’s clinical pillar produced new research methods and new practices. Most health systems/services projects reported new or changed policy or programs. Half of social/cultural/environmental/population health projects (n=7) reported developing a new theory.²⁸

The following examples of project achievement drawn from key informant interviews and case studies demonstrate the diversity of the projects funded by PPSRI and their achievements.

Knowledge Translation to Key Stakeholders

While much of the knowledge translation from PPSRI projects occurred through the traditional channels of peer reviewed publications, presentations or posters at conferences and symposia, and workshops and annual meetings, examples of other approaches to knowledge translation described during the key informant interviews included:

• Requiring all students working on the project to provide a “lay summary” of their research for publication in a pamphlet. Interviewed students reported that while this had been difficult it had provided good training for them;
• Dissemination of information through networks;
• Sitting on advisory boards and providing expert advice to international organizations such as WHO;
• Acting as media spokespeople for the H1N1 pandemic; and
• Requiring all investigators working on their project to participate in conferences and/or symposia.

Partnerships were also an effective way to disseminate information. NPIs reported that partnerships with decision makers and health service providers, or serving on advisory boards had led to PPSRI findings, particularly “real-time findings,” having a direct impact on policy development and service delivery. The ability to form partnerships was underlined by the H1N1 pandemic through the need to rapidly move research findings and expert advice into decision-making venues. NPIs reported that 18% of their projects had resulted in changed policies or programs.

Translating results to the broader public seemed to be somewhat dependent on researchers pre-existing aptitudes and connections. Some PPSRI researchers also had key roles as spokespeople during the H1N1 pandemic and used the media as a channel for disseminating information and research findings. For example, an interviewed researcher reported that his and their colleagues’ membership on different immunization committees in Canada and the US facilitated communication with the public.

Many of the interviewed researchers emphasized that knowledge translation was ongoing. Some researchers interviewed reported having received grants that focused on knowledge translation which contributed to successful dissemination of results.

A few researchers interviewed and surveyed cited a lack of infrastructure as hindering effective knowledge translation and 5% of surveyed researchers suggested specific improvements in this area, such as making data sharing between research institutions mandatory or producing a regular newsletter to report on projects’ progress and findings.

Stakeholders involved in the PPSRI-funded projects

In the mid-term evaluation,²⁹ 66% of researchers reported that research users were directly involved in their projects and a further 21% indicated that research users would eventually become involved. In the present evaluation, 88% of all researchers reported they had worked on projects involving stakeholders and 45% had listed stakeholders in their grant applications (Table 3-2).

Those stakeholders who were involved in PPSRI projects as formal partners (i.e., listed in the grant application) appeared to be involved in all or most aspects of the projects. Stakeholders provided valuable input to projects through the provision of data, in-kind support, access to blood samples, and access to health services and systems. One research team reported being the first team to have access to data from Mexico City about the H1N1 pandemic. Generally, stakeholders were most commonly involved in data collection and project implementation (Figure 3-1), particularly so for health system and health care practitioners.

Figure 3-1: Nature of Stakeholder Involvement in PPSRI Projects as Reported by Researchers

Figure 3-1 long description

Source: NPI online survey QB2 and QD2 and COPI/OPI online survey QB2; Select all that apply type of question.

Note: Data were combined for up to two NPI projects. COPI/OPI reported generally on all the projects they had been involved in.

Influence of PPRSI-Funded Research on Stakeholders

Survey results show that health system and health care practitioners were the group who benefitted the most from PPSRI projects (Figure 3-2). One interviewed researcher’s project created a 10-hour course on pandemic preparedness for hospital workers with a focus on anticipating and coping with an outbreak to decrease interpersonal problems on the front lines and increase a sense of self-efficacy. Moreover, researchers in the field of epidemiology (modeling the spread of disease) attributed effective pandemic responses by practitioners and decision makers to the data their projects had generated.

Figure 3-2: Stakeholders Reported as Influenced to Some/Great Extent by PPSRI Projects

Figure 3-2 long description

Source: NPI online survey QB3 and QD3 and COPI online survey QB3.

PPSRI-Funded High Impact Research Projects

The following two examples illustrate the broad impact generated by PPSRI-funded research. The impacts were captured by conducting case studies of two PPSRI funded projects which included interviews with researchers, partners, research users and trainees involved in the research projects.

4. Collaboration & Partnership

Fostering collaborations and partnerships was a central goal of the PPSRI. Through Team Grants, and both the Seed and Collaborative Research Project Grants under the International Opportunity Program, CIHR–III strived to bring research communities and stakeholders together to create innovative approaches to address important research questions.

CIHR – III collaborative activities and partnerships

Over the life of PPSRI, CIHR–III collaborated with a wide variety of stakeholders to fund research in the identified strategic areas, and to organize a series of workshops and meetings to promote the initiative. Stakeholder organizations included: the Alberta Heritage Foundation for Medical Research (now Alberta Innovates – Health Solutions), the Canadian Food Inspection Agency (CFIA), Canada's Research-Based Pharmaceutical Companies (Rx&D) Health Research Foundation, the Public Health Agency of Canada (PHAC), CIHR Ethics branch, CIHR Institute of Aboriginal Peoples’ Health (CIHR-IAPH), CIHR Institute of Health Services and Policy Research (CIHR-IHSPR), CIHR Knowledge Synthesis and Exchange branch, and the National Natural Science Foundation of China.³¹

A series of workshops and meetings (Table 4-1) were organized by CIHR-III, involving various stakeholders and pandemic experts from different research settings across Canada. The goal of these meetings was to help researchers share their research findings, identify gaps in research knowledge, and also to foster networking and collaboration. The purpose of the application development workshops was to assist researchers in gaining an understanding of the application process and the key components of the application. The majority of researchers who participated in these workshops and who completed the workshops’ evaluation forms rated the overall effectiveness and usefulness of the workshops as good, very good or excellent.

Over the course of the initiative CIHR-III attracted 44% of the initiative financial resources from external partners. Table 4-2 presents the financial commitments for each of the PPSRI’s partners.

The mid-term evaluation reported that nine organizations had expressed an interest in partnering with CIHR on this initiative, and concluded that CIHR had been effective in developing solid and productive partnerships with national agencies. Partner stakeholders interviewed for the mid-term evaluation viewed the partnerships very positively because they enabled them to support research in identified priority areas and to capitalize on the CIHR’s peer review system. The PPSRI was regarded as effective in the production, coordination and integration of national research programming, as well as avoiding duplication.³² Although representatives from four partner agencies were interviewed as part of this evaluation, it was difficult to obtain more detailed perspective on the partnerships as in most cases the interviewed representatives had not been involved in the early stages of the partnership development. However, partners were generally very positive about CIHR’s effectiveness in administering the competition process, managing the grants, coordinating pandemic research nationally, the value of the PPSRI in establishing a research infrastructure, and the expectation that the research had or would create new knowledge. Some of the suggestions mentioned by partners were to have up to date information about the achievements of PPSRI projects, and synthesized information for decision makers.

PPSRI-funded researchers collaborative activities

PPSRI-funded researchers were also encouraged to collaborate with other researchers or different stakeholders through their projects. The survey, the interviews and the bibliometric study provide evidence of the extent of funded researchers’ collaborations.

Survey results show that almost all NPIs (85%) and COPI/OPIs (82%) considered that the PPSRI grant had helped them or their teams to collaborate with other researchers involved in influenza and pandemic preparedness research. Many collaborations were formed during PPSRI funding and have continued following the end of the grant (NPIs 83% and COPI/OPIs 67%). Collaboration was also increased by the H1N1 pandemic, which brought together researchers, decision makers and front-line staff to respond to the threat.

In interviews, NPIs reported that the CIHR workshops and meetings had provided valuable opportunities for networking and collaboration. For some researchers the workshops were an opportunity to exchange data and keep up-to-date on developments of the PPSRI projects at other institutions outside their immediate research area and across disciplines. One researcher reported that networking led to collaboration with another researcher half-way across the country. Both researchers were able to leverage their own local human resources and laboratory capacities, and ultimately produce joint conference presentations and publications. Though not all researchers interviewed reported attending workshops, all were aware of the opportunities.

In the mid-term evaluation, 67% of NPIs and 59% of COPI/OPIs considered their work to involve researchers from areas or disciplines that were outside of influenza and pandemic preparedness work before, suggesting an increase in new collaborations.³³ The percentage of researchers crediting the PPSRI with increasing their international collaboration has increased from 35% of NPIs and 25% of COPI/OPIs in the mid-term evaluation to 68% of NPIs and 43% of COPI/OPIs in this evaluation (Table 4-3).

Successful collaboration with public health agencies was a common theme in interviews with researchers. National and provincial agencies acted as conduits for knowledge translation, sources of feedback, and linkages to the broader research community. One researcher reported the compilation of project data by provincial agencies was critical to their project. Another NPI reported that in light of the H1N1 pandemic, their research team altered the goals of their project to better align with public health priorities, and designed studies based on the feedback they received. This researcher also reported that public health agencies acted as a bridge to international agencies and national agencies in other countries, which facilitated collaboration and the development of common protocols.

National and international collaboration rates

The bibliometric analysis was also used to assess the extent of researchers’ collaborations both nationally and internationally by looking at the number of papers published by PPSRI-funded researchers in collaboration with at least one other researcher from a different institution or from a different country. The findings indicate that the proportions of papers published by PPSRI-funded researchers in collaboration with other researchers are higher than Canadian averages or the world averages for pandemic research at both national and international levels (Table 4-4). The bibliometric network analysis revealed that the following Canadian institutions are the most central nodes of collaboration in the area of pandemic research: University of Toronto, Hospital for Sick Children – Toronto, Mount Sinai Hospital-Toronto, and the University Health Network.

5. Leveraging additional funds

Leveraging additional funds is another indicator used to measure the success of funded projects, the assumption being that the additional funds will contribute to the continuation or expansion of the research projects. Nearly half of the NPIs (45%) and close to one-quarter (24%) of the COPI/OPIs reported that their involvement in their PPSRI grant had helped them to secure additional funding (Table 5-1).³⁴

Sources of additional funding varied, with the most commonly reported source being provincial government organizations. However, the most money was leveraged from industry. Five researchers reported having leveraged additional funding from industry sources. One researcher leveraged 12 grants and/or awards from industry with a total value of over five million dollars. The other researchers reported leveraging between one and four industry grants and/or awards with total values ranging from $300,000 to $500,000.

As well as the sources and value of additional funding leveraged by PPSRI researchers summarized in Table 5-1, researchers reported leveraging resources in the form of in-kind support. For example, three researchers interviewed reported receiving in-kind support from public health providers and other partners, including one researcher who ultimately leveraged support from four multinational pharmaceutical manufacturers.

Fund-leveraging appeared to be constrained by several factors. In an interview, one researcher explained that the sources of additional funding were limited and commented on the ethical implications of obtaining funding from industry. Another interviewed researcher commented on the credibility that having non-industry sourced funding provided to his work. A common theme among interviewed researchers was a competitive market for dedicated public funding towards influenza and vaccine research. Examples included reports that:

• Influenza researchers have difficulty competing with virologists investigating other viruses such as Hepatitis C, Herpes and HIV;
• There are other sources of funding available but, aside from industry sources, none compare in value to those offered by CIHR as part of the PPSRI; and,
• Receiving industry funding can affect perceptions of credibility and objectivity within the public health universe.

6. Program Design

The starting point for the design of the PPSRI was the Influenza Research Priorities Workshop organized jointly by CIHR-III, PHAC and the Canadian Association for Immunization Research and Evaluation (CAIRE) in September 2005. Following the workshop, the CIHR-III, in consultation with its Institute Advisory Board (IAB) and key members of the research community, formed a Task Group to identify the research priorities to focus on, as well as the funding mechanisms to be used. The priorities identified are described in more detail in the Program Background (Section 8) of this report, but here should be underlined that these priorities were validated through consultation with the Canadian Rapid Research Response Team (C3RT) and other stakeholders in pandemic-related fields.³⁵

Beginning in fall 2006, PPSRI offered several types of funding opportunities: catalyst grants, team grants, operating grants, development and planning grants. A total of 31 funding competitions were launched through PPSRI, generating 92 funded projects. Some of these funding opportunities were offered through CIHR’s ongoing slate of programs such as priority announcements, others were offered in partnership with other CIHR institutes and other organizations. Overall most of the funding opportunities included a relevance review process of a letter of intent. Partner organizations were included in the relevance review process to ensure that they could identify applications that aligned with their priorities. A description of the funded projects is offered in the Program Background, Appendix 1.

To assess the effectiveness of PPSRI’s design and delivery the evaluation explored initiative’s strengths and challenges as they were perceived by the funded researchers, as well as different aspects related to grant application and decision process.

Strengths of PPSRI

For NPIs, the main strength of PPSRI related to the infusion of funding targeted to influenza and pandemic research (Table 6-1). Approximately one-third of NPIs noted specific aspects of the grant mechanism such as the design and timelines as strengths. In the key informant interviews, one NPI noted the advantage of being independent from industry funding. PPSRI was also seen as broadening the scope of what CIHR normally funds and providing flexibility to set and change the research agenda. COPI/OPIs were less involved in the grant application and decision process, and tended to identify the ability to build networks and collaborations as a key strength of the initiative.

Key program strengths, identified during interviews with NPIs, were increased research and response capacity. Researchers appreciated that the PPSRI initiative had increased the number of people working on influenza and pandemic research. Multiple researchers reported that the initiative was well-timed and capitalized on public interest in pandemic preparedness. They also credited CIHR for investing in projects which they reported would have been unlikely to attract such substantive funding through normal funding mechanisms.

Challenges and Suggestions for Improvement

The most frequently identified challenge was related to the potential lack of sustainability of the increased researcher capacity and the networks and collaborations that had been developed through the PPSRI. Most NPIs comments related to funding - the need for additional funding, funding renewal, or funding for longer time periods (Table 6-2). The PPSRI grant durations varied from one up to a maximum of three years: catalyst grants were one year, operating grants two years and team grants three years in length.

These findings are consistent with the earlier suggestions made in the mid-term evaluation where key informants also noted the length of the grants as a challenge and suggested that this type of initiative to include a two-year planning and implementation phase followed by five years of funding.³⁶

Other suggestions included involving other national organizations in the competition process and setting up a consortium of manufacturers capable of responding in collaboration with the research community to produce new vaccines.

Aside from the dominant theme of funding-related issues, some interviewed researchers cited access to data and resources as an area in need of improvement. One researcher noted that barriers to data sharing may have been more related to the policies of research institutions than an aspect of the initiative. Another researcher reported that there are a limited number of Level 3 “High Containment” laboratories in Canada, and that H1N1 had been designated a Level 2.5 risk, thus a greater proportion of project funding could be directed toward increasing the number of facilities.

Also consistent with researchers’ comments, interviewed partners suggested the following improvements: the need to sustain the gains in capacity that had been made; involve more industry partners; and increase support for knowledge dissemination and translation.

CIHR partners suggested improving the information available to end-users by ensuring organizations received annual reports.³⁷ One partner noted that while there were opportunities for disseminating information to researchers, information was needed in a form appropriate to decision makers, such as synthesized summaries of key findings by theme. Information would also better meet the needs of partners if partners were more involved in the competition design stages of future initiatives.

Grant Application and Decision Process

NPIs were positive about the grant application decision process (Figure 6-1). While still perceived as effective, an area identified as needing more clarity was the application guidelines. In the interviews, there were also some comments made about timelines and challenges in applying, for researchers who did not have previously established teams or networks.

Figure 6-1: NPI Perceptions about the Application Process

Figure 6-1 long description

Source: NPI online survey QF1

NPIs were also positive about the decision process (Figure 6-2) and few had any suggestions to make about improvements to the process. However, the speed of responses by CIHR staff to requests for information and the transparency of the peer review process were areas where potential improvements could be made. While the overall attitudes of NPIs towards the peer review process were positive, one interviewed NPI noted the challenges associated with finding Canadian peer reviewers for large network grants.

Figure 6-2: NPI Perceptions about the Decision Process

Source: NPI online survey QF2

Figure 6-2 long description

Research Outputs

PPSRI used a variety of funding mechanisms to support research in the pandemic area: catalyst grants, operating grants, team grants, Influenza Research Network (IRN) and other planning and development grants. In order to compare the productivity of these different types of grants, an analysis of research outputs per $100,000 invested was conducted using the survey results (of 51 PPSRI projects) for the following indicators: the number of refereed journal articles, number of research staff and trainees involved in the projects, and the number and amount of additional funding leveraged. This analysis provides insight into productivity of researchers holding different types of grants at a basic level for PPSRI and allows comparability with other similar studies; however, the comparisons between the funding mechanisms should be treated with caution given contextual factors and potential confounds related to differences between funding mechanisms and areas of research. For these reasons, the IRN and other planning and development grants (n=10) are excluded from this comparison due to their specific characteristics and the fact that at the time of the survey IRN project was still active.

As shown in Table 6-3, in terms of productivity per $100,000, overall PPSRI grants produced 1.1 refereed journal articles, involved 3.0 research staff/ trainees and leveraged 0.5 grants/awards. For every dollar invested in PPSRI projects, researchers were able to leverage $1.14 dollars in grants/awards. When comparing by funding mechanisms: catalyst grants produced 1.4 refereed journal articles, involved 7.2 research staff/trainees, leveraged 0.9 grants/awards and for every dollar invested it leveraged another $2.06 dollars in grants/awards. They are followed by operating grants which produced 1.3 refereed journal articles, involved 2.5 research staff/ trainees, leveraged 0.7 grants/awards and for every dollar invested it leveraged another $1.80 dollars in grants/awards. Appendix 3 presents further details for this analysis.

The PPSRI data can be compared with similar analyses conducted for the RMNI and OOGP–Research Reporting System (RRS) data as part of the evaluation of the RMNI conducted in 2012 (Table 6-3). Considering the funding mechanisms characteristics, the most suitable comparison would be between PPSRI and RMNI catalyst grants and PPSRI and OOGP operating grants. Table 6-3 illustrates that RMNI catalyst grants produced 1.2 refereed journal articles, involved 4.3 research staff/trainees, leveraged 1.2 grants/awards and for every dollar invested it leveraged another $5.22 dollars in grants/awards; whereas, OOGP projects funded published 2.8 articles and trained 2.6 research staff/trainees per $100,000. As stated previously, this comparison should be treated with caution due to differences between the programs (PPSRI, RMNI and OOGP), the funding mechanism used (catalyst grants, team grants, operating grants) and different areas of research supported (pandemics, regenerative medicine and nanomedicine, open research). As such, this type of analysis provides a benchmark for future CIHR evaluations and studies.

For PPSRI, further analyses have been completed to compare the productivity and leveraging of PPSRI projects by CIHR pillars and area of research funded. The analysis shows that PPSRI projects funded under health systems and services pillar were the highest in terms of both productivity and leveraging: for every $100K invested, these projects produced 3 refereed journal articles, involved 11.3 research staff/ trainees, leveraged 2.1 grants/awards for every PPSRI grant, and leveraged $3.15 dollars in grants/awards for every PPSRI grant dollar invested (Table 6-1, Appendix 3).

The analysis by area of research found that the highest productivity and leveraging was achieved by the PPSRI projects funded under prevention and treatment area of research: for every $100K invested, these projects produced 2.0 refereed journal articles, involved 7.4 research staff/ trainees, leveraged 1.1 grants/awards for every PPSRI grant and leveraged $2.52 dollars in grants/awards for every PPSRI grant dollar invested (Table 6-2, Appendix 4).

7. Program Relevance

Continued need for Pandemic Preparedness Research in Canada

The present evaluation assessed also the extent to which PPSRI addressed the need for the pandemic preparedness. PPSRI funding has resulted in knowledge creation by projects that would likely not otherwise have been funded. In the absence of PPSRI funding, over two-thirds (68%) of NPIs believed that their project would not have been undertaken at all and approximately one-quarter (28%) felt their project would have been completed on a smaller scale or differently. Very few NPIs (3%) reported that their project would likely have been completed.³⁸ This pattern was similar across all PPSRI theme areas.

Some researchers reported difficulty in getting funding from other sources because their work was at the interface between disciplines, or that they had tried other sources but were unsuccessful. Despite initial difficulties, some NPIs had been able to leverage additional funding as a result of their PPSRI grants, both from CIHR and other sources.

PPSRI was seen as an innovative initiative. One NPI commented that it was good that it had been put in place before the H1N1 pandemic, as the infrastructure and networks that had been established were able to be rapidly mobilized when the pandemic broke out.

The area of research where NPIs and COPI/OPIs thought PPSRI had had the most impact was research into vaccines and immunization (Figure 7-1). This was also the research area in which the most money funded through this initiative had been invested. NPIs and COPI/OPIs had differing views on the extent to which PPSRI had contributed to research on the ethical, legal or social aspects of influenza and pandemic research. Of note is the relatively high proportion of NPIs and COPI/OPIs who did not know the extent of PPSRI’s contribution to research as a whole. This may be because the dissemination of knowledge from projects is ongoing and/or that researchers are not aware of PPSRI contributions outside of their own field.

Figure 7-1: PPSRI’s Contribution to Influenza Research and Pandemic Preparedness in Canada

Figure 7-1 long description

Source: Perceptions of Contribution - NPI Online survey QF4 and COPI/OPI online survey QC2
Percentage of Invested dollars - Based on CIHR data only for NPIs who completed the online survey

These findings are consistent with the results of the 2009 mid-term evaluation for the researchers’ ranking of the pandemic research priorities for the following five years at that time. This supports the idea that the research priorities identified by the PPSRI Task Group were relevant for the field of pandemic research.

The PPSRI increased researcher capacity, formed networks, and established collaborations as shown in sections 2 and 4. It is not clear how many of these would have been formed without the PPSRI or how many will be sustained in the long term. However, there is evidence that there is now a larger pool of trained people who can be called on in the event of another pandemic.

In interviews, one researcher noted that in the time since the H1N1 pandemic, there had been a loss of focus on influenza and pandemic research possibly as a result of “fatigue” with the topic. Others emphasized that, although some progress had been made, there was a need to continue funding for research in this area and that the PPSRI needs to be an ongoing initiative.

Alignment with government priorities and departmental strategic outcomes

CIHR’s mandate is to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health-care system."³⁹ Through the research it funds, CIHR helps to respond quickly and effectively to health crises such as outbreaks of infectious diseases, by rapidly mobilizing researchers to address different area of health research. PPSRI responds directly to the CIHR strategic directions described in the CIHR’s Strategic Plan (2009-2014): Health Research Roadmap by investing in world-class research excellence and addressing health and health system research priorities through preparing for and responding to existing and emerging threads to health.⁴⁰

Alignment with role and responsibilities of the Federal Government

In May 2006, the Government of Canada committed a total of $1 billion over five years to support Canada’s ability to respond to pandemics involving other government departments as Public Health Agency of Canada (PHAC), Health Canada (HC), the Canadian Food Inspection Agency (CFIA) and CIHR.⁴¹ In response to this, CIHR–III one of the 13 virtual institutes of CIHR, that supports research and building capacity in the area of infectious disease and immunity, established PPSRI, with the mission to identify strategic research priorities and to support pandemic preparedness research to address any potential pandemic threat.

In 2010, the Standing Senate Committee on Social Affairs, Science and Technology undertook a review of Canada’s response to the 2009 H1N1 influenza pandemic underlining the importance of pandemic readiness and the need to maintain a focus on pandemic planning. One of the recommendations of this study was that the Government of Canada ensures that future federal pandemic preparedness and response budgets provide dedicated and sustained funding for research (Canada’s Senate, 2010).

All researchers, partners and stakeholders interviewed believed that the federal government had a role in supporting influenza and pandemic research. The reasons cited by interviewees included the following: a pandemic is a national issue; the credibility that federal government funding gives to a research group at the international level; the need for a coordinated agency-wide approach to a pandemic; the federal government’s ability to make data available nationally and internationally; and, the lack of another body with the funding resources or the ability to respond quickly.

References

CIHR - Pandemic Preparedness Strategic Research Initiative – Report on Activities & Outcomes.

Institute of Infection and Immunity (III) – About the Pandemic Preparedness Strategic Research Initiative (PPSRI), Retrieved on May 2012.

Campbell D., Picard-Aitken M., Côté G. et al. (2010). Bibliometrics as a Performance Measurement Tool for Research Evaluation: The case of Research Funded by the national cancer Institute of Canada. American Journal of Evaluation, 31; 66-83.

Goudin, B. (2005) The Impact of Research Grants on the Productivity and Quality of Scientific Research [ PDF (291 KB) - external link ]. Accessed on December 14, 2011.

Graham, I.D., and Tetroe J. (2007). How to translate health research knowledge into effective healthcare action. Healthcare Quarterly, 10 (3), 20-22.

Larivière, V., Archambault, E., Gingras, Y., & Vignola-Gagné, E. (2006). The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities. Journal of the American Society for Information Science and Technology, 57, 997-1004.

Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative, CIHR (May 2009), Retrieved on December 2012.

Moed, H. F. (2005). Citation Analysis in Research Evaluation. Dordrecht, The Netherlands, Springer.

McDavid, J C. and Hawthorne, L.R.L. (2006). Program Evaluation and Performance Measurement: An Introduction to Practice. Thousand Oaks, CA: Sage Publications.

PHAC (2011) - Evaluation of the grant and contribution components of the Avian and Pandemic Influenza Preparedness Initiative - just draft available.

Senate Canada (2010) – Standing Senate Committee on Social Affairs, Science and Technology - Canada’s Response to the 2009 H1N1 Influenza Pandemic [ PDF (465 KB) - external link ], Retrieved on December 2012.

Appendices

Appendix 1: Program background

Recognizing the potential threat of a pandemic influenza, in May 2006, CIHR-III established PPSRI to address the need to develop a coordinated and focused research effort and build research capacity in the field of pandemic influenza in Canada. The mandate of the PPSRI was to identify strategic research priorities and support pandemic preparedness research. PPSRI was guided by PPSRI Task Group formed by researchers who carried out pandemic research and research users who would use the knowledge generated through this initiative.

The following PPSRI research priorities were identified by the PPSRI Task Group:

• Vaccines and immunization programs (optimizing the use and efficiency of existing vaccines and development of new pandemic vaccines);
• The virus (understanding the biology of the influenza virus and rapid diagnostics);
• Prevention and treatment (modes of transmission, use of antivirals and alternate strategies for prevention); and
• Ethics, legal and social issues (research in risk communication, prioritization and regulatory approval processes).⁴²

CIHR received $21.5 million over five years (2006-2011) to support pandemic influenza research and CIHR-III managed to attract another $19 million from external partners (Table 8-1).

Funding was awarded to a nominated principal investigator (NPI) who worked on the project alone or with co-principal investigators (COPIs) and other principal investigators (OPIs). Other researchers as well as highly qualified personnel (HQP) such as graduate students, postdoctoral fellows and research associates may also have been involved in supported projects as they developed.⁴⁴

The following types of grants were launched through PPSRI⁴⁵:

Catalyst grants

27 projects

12% of budget

Catalyst grants served as a first step towards more comprehensive funding opportunities. They supported original projects with the potential to generate high impact results, such as new research tools, inventions, or methodologies. These grants had an average duration of 1 year, and an average value of $190,701.

Operating grants

30 projects

16% of budget

Operating grants helped develop and maintain Canadian health research capacity by supporting projects or teams in all areas of health. They contributed to the creation, dissemination and use of health-related knowledge. These grants had an average duration of 2 years and an average value of $219,887.

Team grants

10 projects

31% of budget

Team grants supported collaborative, multi-disciplinary research teams. Their goal was to enhance understanding of interactions between different areas of health, address major health research questions, and foster the application of knowledge. These grants had an average duration of 3 years and an average value of $1,293,034.

Influenza research network

33% of budget

Three years project (2009-2011) launched in collaboration with the Public Health Agency of Canada (PHAC), to conduct influenza and pandemic preparedness research, foster knowledge exchange between researches and decision makers and train the next generation of researchers. It involved over 130 researchers from different institutions. Grant value of $13,974,459.

Other development and planning grants

24 projects

8% of budget

Other grant programs under which PPSRI projects were funded include the China-Canada Joint Health Research Initiative, two Applied Public Health Chairs, the International Opportunities Program – Development and Planning Grant and other planning and collaborative research grants. With the exception of the two Applied Public Health Chairs which have a 4 and 5 year duration, the average grant duration is 1.4 years, with an average grant value of $131,510.

Figure 8-1: PPSRI’s Contribution to Influenza Research and Pandemic Preparedness in Canada

Figure 8-1 long description

Appendix 2: Evaluation Methodology

To ensure that the study findings were robust, credible, and consistent with Treasury Board Secretariat policy and recognized best practices in evaluation (McDavid et al., 2006), this evaluation used multiple lines of evidence, involving both quantitative and qualitative data. The PPSRI evaluation framework which depicts in detail the indicators, methods, and sources of data collection employed for each evaluation question is presented at the end of this section.

Data Collection and Analysis Activities⁴⁶

Document review and data analysis

CIHR-III documents related to the mid-term evaluation as well as the reports on PPSRI activities and outcomes (available on CIHR-III website) were consulted for this evaluation. CIHR Electronic Information System (EIS) data analysis was conducted when necessary.

On line/telephone survey

All PPSRI-funded PIs and COPIs/OPIs were invited to complete an on line survey. Telephone follow-ups were conducted to remind researchers to complete the survey or assist them in completing the survey over the phone. The short questionnaire captured information about: project outputs, capacity building, knowledge translation, partnerships and collaboration, and general attitudes towards the PPSRI. Responses were received from 40 NPIs and 108 COPI/OPIs providing an overall margin of error of +/-6.1% (Table 9-1).

Key-informant Interviews

Eighteen in-depth interviews with key informants were conducted. The key informants were asked to answer somewhat different questions due to the nature of their roles, therefore unique interview guides were devised for each group:

• Eleven funded researchers were asked about project outcomes, capacity building, partnerships and the PPSRI application and decision process;
• Four partners/stakeholders and three task group members were asked about topics relevant to their role within the program, and to provide an overview of the program.

Case Studies

Two case studies were conducted with PPSRI-funded projects in order to better understand the accomplishments that were achieved through the PPSRI-supported grants, and the factors that contributed to the projects’ successes. Case studies involved interviews with the NPIs, partners/research users and trainees involved in the projects. Eight interviews were conducted to complete the two case studies.

Bibliometric analysis

The bibliometric study provided data on the scientific productivity and impact of the funded PPSRI researches (N= 345, including 61 NPIs and 284 COPs), compared with other Canadian researchers and other researchers publishing in the top most productive countries in the pandemic field for the period 2001-2010. The search was done using the words “influenza” and “pandemic*” in their titles to search the Thomson Reuters’ Web of Science (WoS) and the US national Library of Medicine PubMed database, as well as all the papers published in the journal Influenza and other Respiratory Viruses. Indicators used for this analysis were: Average of relative citation (ARC), Average of relative impact factor (ARIF), Specialization Index (SI), collaboration rate and networking analysis.

Limitations

One challenge encountered in data collection was obtaining information from PPSRI partners through the key informant interviews. Due to personnel changes in partner organizations, many of the partners contacted had not been involved in the early stages of the project. Others had not received CIHR reports and so had limited awareness of the project’s achievements.

In order to assess the impact of PPSRI which had a relatively short life span, 2006-2011, and considering that this evaluation was conducted right after the initiative ended, some of the research outcomes and impacts (e.g. publications, citations) may not be fully covered, as they are ongoing.

Appendix 3: Research outputs per $100,000 invested - Comparison of PPSRI, RMNI and OOGP by type of funding mechanism

Appendix 4

Date modified:

2014-07-16