Evaluation of the Drug Safety and Effectiveness Network Program

Table of Contents

• List of Common Acronyms
• Executive Summary
• Management Response and Action Plan
  • 1. Evaluation Purpose
  • 2. Program Profile
  • 3. Program Design, Delivery and Efficiency
  • 4. Program Performance
  • 5. Relevance
  • 6. Conclusion and Recommendations
• Appendix A: Evaluation Questions and Methodology
• Appendix B: DSEN Logic Model
• Appendix C: Life Cycle Approach Model
• Appendix D: DSEN Organizational Chart (13–03–2013)
• Appendix E: DSEN Query Process Map
• References

List of Common Acronyms

CADTH – Canadian Agency for Drugs and Technologies in Health

CIHI – Canadian Institute for Health Information

CIHR – Canadian Institutes of Health Research

CNODES – Canadian Network for Observational Drug Effect Studies

DSECT – Drug Safety and Effectiveness Cross-Disciplinary Training

DSEN – Drug Safety and Effectiveness Network

DSEN CO – Drug Safety and Effectiveness Network Coordinating Office

DSEN SC – Drug Safety and Effectiveness Network Steering Committee

FCSAP – Food and Consumer Safety Action Plan

F/P/T – Federal/Provincial/Territorial

GoC – Government of Canada

HPFB – Health Products and Food Branch (Health Canada)

HC – Health Canada

INESSS – L'Institut national d'excellence en santé et en services sociaux

KT – Knowledge translation

MHPD – Marketed Health Products Directorate (Health Canada)

OOGP – Open Operating Grant Program

PAA – Program Alignment Architecture

PHAC – Public Health Agency of Canada

PMDSE – Post market drug safety and effectiveness

RRS – Research Reporting System

SPB – Strategic Policy Branch (Health Canada)

STIHR – Strategic Training Initiative in Health Research

TBS – Treasury Board Secretariat of Canada

Executive Summary

The Drug Safety and Effectiveness Network (DSEN) was established by the Government of Canada in 2008-09 as one of a number of initiatives that contribute to meeting the objectives and goals of the 2007 Food and Consumer Safety Action Plan (FCSAP).^{Footnote 1}

DSEN generates and promotes the use of evidence on post-market drug safety and effectiveness (PMDSE) and contributes to increasing capacity to undertake quality PMDSE research. The DSEN program links existing research centres in PMDSE across Canada. The DSEN Coordinating Office is located within the Canadian Institutes of Health Research (CIHR).

As the first evaluation of DSEN, the evaluation focused on assessing the design and delivery of DSEN, measuring progress toward the achievement of its immediate expected outcomes, and identifying areas for improvement. It was agreed that the scope of the evaluation would not include a review of the impacts of a policy and decision-making support enterprise in a research agency. The evaluation covers the time period from the creation of DSEN in fiscal year 2008-09 to end of fiscal year 2012-13 and meets the requirements of the Treasury Board’s Policy on Evaluation (2009) and the Financial Administration Act.

Key Findings

• DSEN and its Coordinating Office (DSEN CO) have established Canada’s first national PMDSE network within the planned timelines and made advances in developing and establishing management and structural protocols and procedures to achieve its expected objectives.
• The DSEN CO has developed DSEN Steering Committee (SC) Terms of Reference, signed formal agreements and developed guidance documents. There is, however, evidence that some primary stakeholders are not yet entirely clear on the role and function of DSEN and their relevant roles and responsibilities within the program. Specifically, there are three aspects of the program which require clarification: the roles and responsibilities of the DSEN Steering Committee; the time taken to prioritize and respond to queries; and the level of independence of DSEN’s researchers.
• The establishment of DSEN has resulted in the creation of a national PMDSE forum and has brought greater coordination to PMDSE-related research activities. According to interviewees, this has been achieved through better leveraging of expertise across various groups of researchers, developing PMDSE capacity in Canada, and fostering culture change by encouraging an increased willingness among researchers to share preliminary results with decision-makers.
• The DSEN CO has established seven methodological-specific research teams and received a total of 53 Queries of which 36 have been prioritized for research during the period under review. Of the 36 prioritized Queries, research is complete for seven (12 Queries were completed as of September 2014), is ongoing for 22, and is planned pending resources for the remaining seven Queries. The DSEN program has distributed a total of 101 grants and awards, with 62 of these being used to support projects related to prioritized Queries and 39 used to support networking, knowledge translation and capacity development. For the period under review, there is limited evidence of the extent to which DSEN research has informed decision-making needs; however, this should be considered in the context of the relatively recent launch of the DSEN program, the small number of completed Queries and the time required for the response to a Query to influence decision-making.
• Based on the seven completed Queries, DSEN research has provided evidence to inform a few PMDSE decisions, including:
  • The identification or confirmation of a health risk and making recommendations to final labeling for a drug.
  • Informing a policy decision.
  • Providing evidence for the Common Drug Review to inform a recommendation.
• While most interviewees expressed opinions about the importance of the timeliness of the delivery of evidence and the need to track timeliness, the program does not systematically collect data on the time to taken to respond to Query submissions.
• Due to an inability to separate out the DSEN component of Health Canada (HC) expenditures related to Targeted Oversight, the delivery costs were estimated for the period from 2008-09 to 2012-13, with the exception of fiscal year 2012-13. Over this period, the estimated total delivery costs of DSEN were 33% ($8,656,676/$25,275,571) of total program expenditure and decreased from 100% in 2008-09 (the first year of operation) to 20% in 2012-13. The DSEN CO and HC related delivery costs were approximately equal over this period: 16% ($4,266,169/$25,909,821) and 17% ($4,390,507/$25,909,821) of program expenditures, respectively.
  • For fiscal year 2012-13, the delivery costs of the DSEN CO were 10.4% ($1,129,949/$10,808,622) of total expenditures. The delivery costs for the DSEN program, including both DSEN CO and HC expenditures were 19.5% ($2,116,292/$10,808,622) of total expenditures.
  • The 2012-13 cost data can be used as a baseline for the program for future analyses within the Treasury Board allocations for operating and grants and awards expenditures; however, there is a need to improve the tracking of DSEN expenditures across program activities, such as capacity building and knowledge translation, to better report on program costs.
• The DSEN program is working to address the continued need for the active surveillance of drug safety and effectiveness in Canada. Its goals and objectives are consistent with the roles and responsibilities of the federal government and align with Health Canada (FCSAP) and the federal governments’ priorities on targeted oversight of health products, as well as CIHR’s strategic directions.

Recommendations

The following recommendations are intended to support DSEN in positioning the program to achieve its objective of better informing pharmaceutical post market drug safety and effectiveness decision-making across the Canadian health care system.

1. The DSEN CO, in consultation with its partners and stakeholders, needs to examine key design and delivery features of DSEN to identify areas where efficiency and effectiveness can be enhanced. Key areas for further examination include:
   1. Clarify key aspects of DSEN’s operation with program stakeholders
      The clarification and common understanding of several aspects of DSEN’s operation would help strengthen the program:
      • Effective formats for communicating research results in response to Queries to stakeholders according to their different mandates and information needs (i.e., knowledge translation).
      • The prioritization of submitted Queries.
      • The appropriate level of independence of DSEN researchers in the context of their interactions with query submitters to define research protocols in response to Queries.
   2. Establish service standards for the Query submission and response process to clarify expectations for Query submitters and support performance measurement.
      • Decision makers require clear and transparent timelines for the delivery of research results in response to Queries to make timely decisions on the safety and effectiveness of marketed drugs.
      • There is a need for improved dialogue between Query submitters and DSEN researchers at the outset of submission to clarify the information required to respond to the Query and the context and timelines of the decision-making process the research result will inform or support.
      • The DSEN CO should develop and implement service standards, such as target response times, to enable submitters and research teams to establish common expectations and agreement on timelines and milestones, taking into consideration the methodology and scope of the proposed research, at the outset and throughout the Query submission and response process.
2. The DSEN CO, in consultation with Health Canada and CIHR, should review the current performance measurement strategy to identify changes to better monitor performance against expected outcomes.
   • The DSEN CO should identify the indicators to collect and track information to better monitor, assess and communicate the performance and impact of DSEN. In particular, additional indicators should be developed relating to timeliness, program expenditures, indirect training and capacity development, and the longer term benefits of query responses on the Canadian health care system.
   • In terms of program expenditures, DSEN needs to track and report on expenditures in greater detail in order to: separate out DSEN related expenditures from the broader Targeted Oversight activities of FCSAP; and, map the use of grant funds by research teams to support PMDSE research, capacity development, knowledge translation and network activities.

Management Response and Action Plan

1. Evaluation Purpose

1.1 Evaluation Purpose, Issues and Scope

The purpose of the evaluation was to assess the program’s design and delivery, performance and relevance to date through a targeted study. Subsequent to the onset of the evaluation, the scope was expanded to ensure that the evaluation would meet TBS evaluation coverage. The evaluation assessed the extent to which the current delivery model facilitates the achievement of the program’s outcomes to identify potential improvements; however, it was agreed that the scope of the evaluation would focus on assessing the achievement of immediate outcomes and not include an assessment of the impacts of a policy and decision-making support enterprise in a research funding agency. The issues and questions addressed by the evaluation were developed based on program considerations and information needs and aligned with core issues outlined in Treasury Board’s Policy on Evaluation and Directive on the Evaluation Function (see Appendix A). The evaluation meets the requirements of the Financial Administration Act and the Treasury Board’s Policy on Evaluation (2009) and covers the activities of the DSEN program from its creation in 2008-09 to the end of fiscal year 2012-13.

1.2 Methodology

In line with Treasury Board’s Policy on Evaluation and recognized best practice in evaluation, a range of methods - involving both quantitative and qualitative data - were used to triangulate evaluation findings (see Appendix A for a full description of the data collection methods used). The data collection cut-off point was March 31, 2013.

1.2.1 Document Review

The review of documents provided information on the history and objectives of the DSEN program as well as insight into changes in the program’s implementation and activities, which contributed evidence addressing some of the evaluation questions. In addition, the document review assisted with the development and formulation of the interview questions. Some of the documents reviewed included: background documentation related to the development of DSEN; Senate reports; peer-reviewed journal articles; and, reports and presentations.

1.2.2 Review of Research Reporting System Progress and Annual Reports

The Canadian Institutes of Health Research (CIHR) formally launched the Research Reporting System (RRS) instrument on March 31, 2011.^{Footnote 2} CIHR requires Nominated Principal Investigators (NPIs) to report on their CIHR-supported research results via the RRS. The intention is to use these RRS reports for a variety of internal and external purposes in order to obtain better evidence on the effectiveness of CIHR funding programs, advance CIHR's Knowledge Translation mandate, and contribute evidence to enhance CIHR’s accountability within the Federal Government and to Canadians for their investment in health research. CIHR also uses the RRS data for a variety of internal uses, including how to better manage the process of funding health research.

All NPIs on Open Grants and selected Priority Announcements, where the authority to spend funds expires as of March 31, 2011 or later, must submit end of grant reports via RRS. NPIs have 18 months after the end of their grant period to complete their reports, and CIHR provides on-going support to assist NPIs in this task.

The DSEN program was an early adopter of the RRS reporting system. This evaluation uses CIHR’s RRS as one of its lines of evidence. The RRS reports provided data on the outputs and outcomes of both completed research projects from the first cohort of funding and the outputs and outcomes of the seven DSEN supported methodological teams as reported in their first annual reports which also made use of the RRS instrument. These reports were received in the 2012-13 fiscal year.

It is important to note that the submitted RRS responses used by this evaluation are not representative of the overall number of DSEN-related grants and awards, and the following reported data should be interpreted with caution. A limited amount of time has elapsed since program inception and only the 2009 DSEN catalyst grant recipients have completed their projects. This has resulted in not only the availability of a limited number of RRS reports for DSEN/CIHR-supported projects where funding has expired (10), but also only a limited number of Queries being completed (seven) by March 31, 2013.^{Footnote 3} A further limitation is that the RRS reports completed by DSEN researchers, as well as DSEN’s seven teams’ progress reports, are abbreviated versions of the full RRS instrument, with a reduced set of questions.^{Footnote 4}

Finally, it is important to note that information in the RRS reports is not directly indicative of the responsiveness of the program to its primary stakeholders who submit research questions for address by DSEN.

1.2.3 Key Informant Interviews

A total of 17 interviews were completed with key informants and provided information to address all of the evaluation questions. The interviews were structured to address issues that other lines of evidence could not, or had limited ability to, address. Table 1.1 presents the breakdown of interviews by the three respondent groups.

1.2.4 Case Studies

Two purposively selected case studies were conducted involving a total of six interviews as well as document review to support the background context of the case studies. For each case study, one of the lead researchers was interviewed along with two decision makers engaged with the development of the research question (Query). In addition to interviews, the case studies were informed by a review of the query submission, research report as well as other information provided by the key informants. Table 1.2 presents the breakdown of interviews by the two respondent groups.

1.2.5 Citation Analysis using Google Scholar Data

Raw citation data was used as a proxy to determine the “quality” of the evidence generated or updated in the peer-reviewed journal articles published by DSEN researchers. Google Scholar was used to collect this raw citation data on the approximately forty articles published by DSEN-supported researchers that were related to funding received by the program.

1.3 Limitations

It is important to highlight several limitations, including:

• There has been a limited amount of time since program inception, with the seven DSEN research teams created in 2011, which may have impacted the program’s ability to achieve many of its expected outcomes. This was mitigated by assessing progress towards immediate outcomes, rather than attempting to measure intermediate and long-term outcomes.
• The RRS reports available for DSEN-related grants were abbreviated versions of the full end of grant reports and included a limited amount of information regarding the outputs and outcomes of the grants. This was mitigated by including other lines of evidence, including:
  • Case Studies: these provided additional evidence of outcomes related to the outcomes of research projects; and
  • Financial Reporting Documents: these were used to assess the extent of indirect capacity development DSEN’s research teams provided.
• The reporting and tracking of financial information was not uniformly implemented across the DSEN CO and DSEN related activities at HC. As a result, the TBS allocations were used as a proxy for actual expenditures where DSEN specific spending could not be disaggregated from HC spending on Targeted Oversight activities under the FCSAP. While not preferred, it is expected that any variance would not be material.
• While several members of the DSEN Steering Committee were replaced post data collection phase and not all invited DSEN Steering Committee members agreed to be interviewed, a majority of relevant SC members (i.e., sitting members at the time of data collection) were interviewed.

2. Program Profile

2.1 Overview of the DSEN Program

The DSEN program was established by the Government of Canada in 2008-09. The DSEN is one of many initiatives that contribute to meeting the objectives and goals of the 2007 Food and Consumer Safety Action Plan (FCSAP). DSEN generates and promotes the use of evidence on post-market drug safety and effectiveness (PMDSE), and contributes to increasing the capacity to undertake quality PMDSE research within Canada. The DSEN program is a virtual network that links and facilitates coordination among existing research centres in PMDSE across Canada, with its Coordinating Office located within the Canadian Institutes of Health Research (CIHR).

DSEN is a health portfolio initiative, funded by the Government of Canada (GoC) through Health Canada (HC) and CIHR. The DSEN program was allocated $32 million for the period from 2008 to 2013, and ongoing funding of $10 million per year. DSEN has distributed a total of $17,253,145 in grants and awards from FY 2009-10 to 2012-13 (Figure 2.1).^{Footnote 6}

Figure 2.1: Grants and Awards Expenditure by Fiscal Year

Long description: Grants and Awards Expenditure by Fiscal Year

Table 2.1 outlines the DSEN program’s two primary objectives and key activities for achieving these two objectives. The DSEN Logic Model in Appendix B provides a full description of the program’s activities and expected outcomes.

Source: DSEN Dashboard, November 7, 2012

The evidence that is generated by DSEN-supported research complements existing, as well as new post-market surveillance and reporting activities included under the FCSAP and other Canadian research initiatives. This evidence is intended to enable key healthcare decision makers to better determine safety and effectiveness profiles for drugs and to implement measures that will promote their optimal use. A diagram of the life cycle approach model of pharmaceuticals regulation in Canada and DSEN’s role is provided in Appendix C.

As of March 31, 2013, the DSEN program has utilized a variety of funding mechanisms to support PMDSE-related research, including: catalyst grants, team and operating grants; rapid funding grants; salary awards; fellowships; and, doctoral awards.

In 2011, DSEN established seven methodology-specific research teams (research teams) which receive funds to support network activities, capacity development and knowledge translation activities. One team, the Canadian Network for Observational Drug Effect Studies (CNODES), receives funds to support research activities^{Footnote 7}; whereas the other six teams must apply for additional funding to support the query-related research they undertake on a query by query basis.

The following are identified partners and stakeholders for the DSEN Program:

• Health Canada;
• Officials from federal, provincial and territorial (F/P/T) governments with direct responsibility for making public health decisions which promote the safe, effective and efficient use of drugs;
• Canadian health organizations (e.g., Canadian Institute for Health Information, Canadian Agency for Drugs and Technologies in Health);
• PMDSE researcher community and research organizations;
• Data holders, both public and private (federal/provincial/territorial drug plans; health insurance providers, academic institutions, health care providers, pharmaceutical industry, etc.);
• National associations (e.g., Health Council of Canada) and international organizations (e.g., U.S. Food and Drug Administration; European Medicines Agency); and
• Doctors, health care providers, health care organizations and patients.

Table 2.2 details the status and number of Queries received, as well as the relevant query submitters, as of March 31, 2013.

2.2 Query management process

The DSEN CO has established a formal query management process (Appendix E) with a supporting framework and guidance documentation. Several decision making organizations are currently able to submit queries (e.g., HC, provincial and territorial health bodies, CADTH).

The DSEN CO facilitates discussion of a potential query between the submitting party and its seven funded research teams. These teams then respond, confirming if they are capable of answering it adequately. These discussions lead to the decision makers either formally submitting their final query to DSEN or pursuing another avenue to address it. The final queries are then considered by the DSEN SC for advancement to the DSEN prioritized research agenda.

One of the roles of the DSEN SC is to endorse the results of the query prioritization process to ensure that the most pressing and relevant queries are examined first. The DSEN CO has developed a Multi-Criteria Decision Analysis (MCDA) tool to assist the SC when evaluating and ranking query submissions.

The MDCA tool is a set of weighted criteria that is used to rate feasible queries to be recommended to the DSEN SC for prioritization to the DSEN research agenda. Through extensive consultations between DSEN and provincial and federal representatives, the MCDA tool was developed and piloted by a working group comprised of members from the DSEN CO, HC and provincial health organizations.

As of March 31, 2013, the DSEN SC has accepted all submitted queries, due to the limited number of queries submitted. DSEN expects that the volume of queries submitted by stakeholders will increase as the provinces and other eligible groups gain the awareness, capacity and experience to submit feasible queries. Thus, in time, the SC will consider all new queries and prioritize them based on the available funds.

All DSEN research teams undertaking research in response to a query must develop a proposal for answering the query to be discussed with the query submitter. Once the approach is developed to support the needs of decision makers, the research team submits, as necessary, an application for a rapid funding grant to support the research they will conduct in relation to the accepted query, which is then peer reviewed.^{Footnote 8}

In October, 2012, the Science Advisory Committee (SAC) was formed to help expedite the early stages of query submission by convening preliminary reviews of potential queries. The SAC is charged with providing an informal discussion forum to decision makers and researchers for the development of DSEN Queries. This facilitates the development and submission of scientifically, methodologically, and technically feasible DSEN Queries to the DSEN CO, increases efficiency of the Query management process and enhances the direct interaction and communication between DSEN researchers and decision-makers.

If a proposal is recommended for funding by peer review, the team receives a rapid funding grant to support the research they will conduct in relation to the accepted query. If no proposals are received from the seven DSEN research teams, an open call may be made by the DSEN CO to the general research community, which would also be peer reviewed. All proposals rejected by peer review, as well as queries with no responses, are returned to the submitter, who then decides to resubmit the query or, resolve the query through other methods.

3. Program Design, Delivery and Efficiency

3.1 Key Findings

• The DSEN Coordinating Office (CO) has established Canada’s first national post market drug safety and effectiveness (PMDSE) network within its planned timelines, and has made significant strides at developing and establishing management and performance protocols and procedures to enable its network to achieve its long term objectives.
• The program has managed its finances well and has not gone over its allotted funds.
• Baseline data has been collected on the program’s efficiency that can be used in future efficiency analyses.
• Non-monetary efficiency has improved, in terms of the delay between the submission of a query and the onset of research, reducing the average gap from 24 months in 2009 to 4.5 months in 2013.
• The DSEN CO has made extensive progress in developing the processes and structures necessary for it to deliver on its objectives.
• Although the DSEN CO has developed DSEN Steering Committee (DSEN SC) Terms of Reference, signed formal agreements and developed guidance documents, there is evidence that members of primary stakeholder groups do not fully understand DSEN and their relevant roles and responsibilities.
• It is not currently possible to determine whether there are more cost-efficient methods that could be adopted due to a lack of comparator data.

3.2 Design and Delivery

This section of the report relating to program design and delivery is divided into two parts:

1. An analysis of the existing design and delivery of the DSEN to assess the extent to which the program has been implemented as planned, including the establishment of a monitoring and management framework, facilitators and hindrances, and identified areas of improvement; and,
2. An analysis of the cost of program delivery, as planned and in practice to date.

3.2.1 Governance and the extent to which roles, responsibilities, and accountabilities of the health portfolio partners (HC and CIHR) are clearly defined and followed

The DSEN program is a health portfolio integrated partnership between CIHR and HC. Figure 3.1 details the roles and responsibilities of the three partners. HC’s Strategic Policy Branch (SPB) acts as portfolio secretariat for the program providing policy oversight and monitoring the performance and achievement of outcomes of DSEN, and reports this progress under the FCSAP.

Although the DSEN CO has developed Terms of Reference and other governance-related documentation, there is evidence of some confusion among some DSEN SC members regarding the role and function of the SC, which may undermine its functionality.

The opinions of interviewees were mixed as to whether the current governance structure and responsibilities are working, particularly with respect to the SC. Nine out of 15 SC members were interviewed and some interviewees indicated that they were not satisfied with the present role of the SC, nor with the types of discussions and advice being generated, and expressed doubts on whether their opinions were being heard. These same individuals felt that the SC was not fulfilling its role to provide strategic advice and direction. One SC member felt that the discussions were highly technical in nature while a few others felt that a better balance in discussions between safety and effectiveness was needed. A few members expressed the need for a greater level of transparency throughout the process and of not overloading SC members with information at meetings.

Some interviewees felt that the current membership of the SC was appropriate, while others felt that it did not necessarily have the right mix. For those who questioned the mix of current membership, no suggestions were provided about what they would suggest to be a more appropriate mixture of members.

3.2.2 A Planning and Management Framework has been Established

The DSEN CO has been proactive in establishing and documenting the activities of the network (see Appendix B: DSEN Logic Model; Appendix D: DSEN Organizational Chart; DSEN Steering Committee Terms of Reference^{Footnote 9}) and developing processes for the management of Queries (see Guidance Document for Submitters of DSEN Queries^{Footnote 10}; and Query Submission Process Appendix E). While DSEN is still in the process of evolving, the DSEN CO continues to formalize processes to avoid ad hoc decision-making, and to operate in a transparent manner.

3.2.3 Performance Measurement and Evaluation Framework

A Performance Measurement and Evaluation Framework (PMEF) for the DSEN program was developed by the DSEN CO and partners in collaboration with CIHR/HC, and post-market evaluation experts and approved in August 2010.

The PMEF makes use of CIHR’s Research Reporting System (RRS) data for its annual reporting.^{Footnote 11} Whereas CIHR requires grant recipients to complete an RRS report within 18 months of expiry to use funds, the DSEN CO has initiated a policy that requires DSEN grant recipients to complete their abbreviated RRS reports within six months of expiry to use funds, to expedite the collection of data for reporting purposes.^{Footnote 12} DSEN supported research teams are also required to submit annual RRS update reports electronically.

In November 2012, the DSEN CO presented its DSEN Steering Committee (DSEN SC) members with a draft dashboard which it plans to use for reporting performance measurement data to the committee on an annual basis.

While the program has taken active steps in developing a detailed performance measurement strategy and while most interviewees felt that the established mechanisms kept them informed, a few interviewees suggested that DSEN was overly “bureaucratic” and that more emphasis, in terms of performance reporting, should be on outcomes, rather than process. Furthermore, the data collection activities undertaken by this evaluation have revealed performance measurement gaps that have reduced the ability to accurately determine program performance and, in particular, the efficiency of the program (both in terms of delivery of query responses/evidence and financial). An assessment of the current utility of the data collected by the PMEF to support future evaluations should be undertaken to identify areas in the framework which need improvement.

The findings from this evaluation will help inform how to best revise the PMEF to ensure that it provides the DSEN SC, DSEN program partners and stakeholders with the information needed to make program related decisions and support the strategic priorities of DSEN.^{Footnote 13}

3.3 DSEN Program Costs and Efficiency

3.3.1 DSEN Program Costs

The DSEN program receives an annual budget to support the activities of the network, the DSEN CO, and the DSEN-related activities of HC. Table 3.1 presents the program delivery costs associated with the program from fiscal year (FY) 2008-09 to 2012-13. It is important to note that the FCSAP reporting structure is such that DSEN specific spending by HC is aggregated with spending on other Targeted Oversight activities (e.g., periodic safety update reporting) and, as result, this analysis uses the Treasury Board Secretariat of Canada (TBS) allotments as proxy estimates of DSEN-related expenditures. While not preferred, the approach is acceptable and any variance is not expected to be material.

The proportion of funding spent on program delivery as a whole is higher than other programs delivered at CIHR (Table 3.1) but is largely in line with the percentage approved in the program’s TBS submission. The initial high percentage of delivery costs to total program expenditures can be explained by the resources required to start up a novel research network that was designed to meet and respond to the needs of specific stakeholders, rather than supporting investigator driven research. Resources were used to refine and develop the network’s structure in terms of funding instruments, the type of methodology required to address Queries and the establishment of the seven teams. The percentage has decreased over the period under review; the proportion spent by the DSEN CO and by HC on the delivery of the DSEN is approximately evenly split, with the most recent year’s proportions being 10.6% and 9.3%, respectively.

While data availability restricted the granularity of our analyses, more detailed costing information is available for fiscal year 2012-13. Table 3.2 presents the planned and actual operational expenditures for DSEN for fiscal year 2012-13.

Table 3.3 details the ratio of expenditures dedicated to the administration and delivery of the program. The program realized a retained surplus of $704,480, with the majority of the retained surplus coming from the DSEN CO, including reduced salary requirements (the DSEN CO has one of its allocated full-time equivalent staff (FTE) unfilled due to CIHR’s 2012-13 vacancy management policy, and a second FTE absorbed by CIHR corporate), reduced Operations and Management costs, and reduced salary related costs due to the vacancies (i.e. benefits and accommodation).^{Footnote 14}

In the future, the savings realized by the program (largely related to staffing vacancies both within CIHR and HC) may be reduced if the DSEN CO receives increased numbers of Queries which would require hiring the program’s full allocation of FTEs in order to ensure the quality of the support for network activities and as short of a turn-around time as possible for submitted Queries. Ongoing program monitoring and the next evaluation should examine the impact this may have on the effectiveness of the program.

It is important to note that the $8,692,330 in grants and awards was used to support both the program’s query-related research and capacity development activities. While the DSEN CO has provided funds to support its seven research teams to conduct PMDSE research in Canada, it also invested $3,343,219 in direct capacity development, awarding bridging grants to early career PMDSE researchers and supporting doctoral students through doctoral awards (38.5%). Furthermore, five of the DSEN research teams reported spending $551,292 on stipends to provide training opportunities for students from all levels of postsecondary education (e.g., Bachelors, Masters, Doctorates and Postdoctoral Fellows).^{Footnote 16} These funds came from their respective Team Grants.

Previous evaluations of CIHR programs have also included opportunity costs related to the time peer reviewers spend reviewing for CIHR. Peer reviewers spent, on average, approximately six hours reviewing DSEN-related proposals per Rapid Funding review panel convened (each review panel has three reviewers/readers).^{Footnote 17} This amounts to $372 per reviewer per application (rounded to the nearest dollar), or, a monetized cost of $1,117 per Rapid Funding application (rounded to the nearest dollar). DSEN held review sessions for 13 rapid grant applications in the 2012-13 fiscal year,^{Footnote 18} which represents a total monetized cost of peer reviewer time spent reviewing applications for funding to support prioritized query research of $14,510. It is important to note that this cost is not paid by DSEN, CIHR or HC, but represents a sacrifice made by the peer reviewers.

While the monetized cost of peer reviewer time is negligible in comparison to the overall budget for DSEN and, as such, has a limited impact the efficiency of the program, the costs represent time taken away from the peer reviewers’ primary task — to conduct research (most peer reviewers are PMDSE researchers). This time is in addition to other peer reviewer commitments these individuals may have. Most of the researchers who serve as reviewers for DSEN also participate in other peer review committees for CIHR and other organizations. Therefore, it is important to keep the time reviewers spend on peer review to a minimum, so that they can devote their time to research.

3.3.2 Efficiency of the DSEN Program

The assessment of the efficiency of the DSEN program involves assessing the program’s resource utilization in relation to the production of outputs and progress toward expected outcomes. As outlined in the TBS Directive on Evaluation Function, the assessment of program performance requires the demonstration of efficiency and identifies the need to compare program outputs and outcomes with program expenditures as one way to assess efficiency (see Appendix A).

Previous evaluations of CIHR programs have used the number of applications received by a program in a given fiscal year to determine the cost-efficiency of the program (i.e., program costs/# of applications). This methodology is not appropriate in the context of DSEN because unlike many of CIHR’s programs, such as the Open Operating Grant Program (OOGP) which has two regular annual competitions with application deadlines in March and September, the DSEN program receives applications on an ad hoc basis and holds funding competitions on an as needed basis.^{Footnote 19}

This evaluation estimates the program’s efficiency by looking at the total operational budget for the period between FY 2009-10 to 2012-13 and key outputs of the DSEN CO for the program. The total expenditures for this period were used as the denominator, while the total numbers of Queries received/accepted/answered were used as numerators. The results provide an indication of the costs, to date, associated with the production of indicators (i.e., number of grants and awards, number of Queries received, accepted and answered) related to two key program outputs: funded research activities; and new post market drug safety and effectiveness evidence.

It is also important to note that the costs per output may seem rather high, but that these figures include the costs associated with the start-up of the DSEN program and that its seven supported research teams were only formed over the 2011 calendar year (one was formed in January while the remaining six teams were established in September). Given that, it is expected that subsequent efficiency analyses conducted as part of program monitoring and the next evaluation should reveal a lower cost per output.^{Footnote 20}

In total, the DSEN CO has distributed a total of 101 grants and awards, as of March 31st, 2013. The total cost, per output (grants and awards), was $79,430, with the total cost per query received being $151,367 and the cost per query answered being $1,146,061 (Table 3.4).

It is important to note that the costs associated with Query responses are likely over-estimated due to the nature of the program and the activities undertaken by DSEN CO. Integrated knowledge translation is a fundamental component of the delivery of DSEN and requires regular interactions amongst the DSEN CO, the methodological research teams and research-users to ensure that the research conducted addresses identified information gaps and is disseminated to relevant user groups. The DSEN CO facilitates and tracks the exchange of information and results between the research teams and relevant decision makers as well as hosts regular events (e.g., semi-annual Network meetings) to disseminate research results and to provide further opportunities for discussions between DSEN researchers, decision makers and relevant stakeholders. Integrated KT approaches to research are also required within the research teams. It is expected that the teams contribute to the development of strategies for knowledge translation within DSEN and will continue to participate in KT activities supported through existing and established methods and channels. The allocation of FTE activity related to the extensive KT activities is not currently tracked which prevents the exclusion of these activities and their associated costs in the efficiency calculations presented above.

3.4 Factors Facilitating or Hindering Program Design and Delivery

3.4.1 Innovative Query research funding design

The DSEN program currently uses two primary funding instruments for maintaining its network and supporting query research — team grants and rapid funding grants. Since 2011, DSEN has supported seven teams of researchers with various areas of specialization in different research methodologies of PMDSE.

Each research team receives funds to support networking, knowledge translation activities, and capacity development. The Canadian Network for Observational Drug Effect Studies (CNODES) team receives additional funds to support research activities.^{Footnote 21} The maintenance of these seven teams provides HC and other query submitters a readily available pool of experts that can respond rapidly to emerging drug safety and effectiveness issues.

The program initially used CIHR’s Catalyst Grant funding instrument between 2009 and 2011 to engage PMDSE researchers in responding to Queries until such time as the DSEN research teams could be established. While it was expected that this funding instrument would be more responsive than other CIHR funding instruments in extant, there were substantial delays to the onset of funding, due to the ad hoc nature and timing of query submissions and the time required to launch corresponding catalyst competitions.

After the establishment of the DSEN funded research teams, the DSEN CO subsequently developed its own funding instrument, the Rapid Funding grant, to support the research activities of the six teams that must apply for funds to address Queries on an as-needed basis. The Rapid Funding granting instrument is designed to deliver PMDSE research funds in a more expedient manner as compared to the Open Operating Grant Program or Catalyst grants (see section 4.3. for a discussion of the timeliness of query responses).

To do this, the Rapid Funding grant has more tightly specified application requirements and requires less time for review while retaining the academic rigor of the review process. For example, peer review by teleconference reduces the time it takes for a prioritized query to be researched by one of DSEN’s established research teams. In 2012-13, DSEN CO received and reviewed 16 applications through the Rapid Funding tool, of which nine projects were funded for a total cost of $1.3 million over three years. The DSEN CO continues to fine-tune the review process to strike a balance between reduced application completion burden yet ensuring that applications have sufficient data for the peer reviewers to adequately evaluate the proposals.

3.4.2 Safety versus Effectiveness

Health Canada was the lead for the Treasury Board submission for the DSEN program, in partnership with CIHR. As the lead, HC began to accumulate potential Queries, which were submitted to DSEN in its first year of operation. As outlined in Table 3.5, HC has submitted the majority of Queries to date, with many of the Queries submitted by HC focused on drug safety due to HC’s role to regulate drugs. Table 3.6 presents the focus of prioritized Queries from 2009-10 to 2012-13, with sixteen Queries focused on drug safety, seven focused on safety and effectiveness and eight focused on effectiveness.

The opinions of interviewees were mixed about the appropriateness of the blend between safety and effectiveness of prioritized Queries. Some of the interviewees suggested that the current focus of the studies undertaken by DSEN researchers is appropriate, while other interviewees expressed concern that there is not enough balance between the two types of Queries. Furthermore, one interviewee expressed concern that the MDCA tool developed by the DSEN CO may favour the prioritization of effectiveness studies over safety studies. Therefore, it would be of benefit to re-examine the MDCA tool to determine if the tool could be revised or if another approach is applicable. The DSEN CO is aware of this issue and is currently discussing the review of the MDCA tool with HC and other stakeholders.

3.4.3 Funding Mix

While interviewees agreed that DSEN must fund both post market research and capacity/infrastructure development, there was no agreement on what should be the appropriate distribution of funds. Based on data for the 2012-13 fiscal year, the current distribution of funds between post-market research and capacity/infrastructure development is approximately two-thirds (62% or $5,349,111) to one-third (38% or $3,343,219), respectively; it is expected this ratio will likely increase in favour of research support as more Queries are submitted and prioritized.^{Footnote 22} It is important to note that the research funds given to the CNODES team are not included in this calculation because the research funds for CNODES were included in their initial Team Grant and were distributed in 2010-11. The addition of these funds would increase the ratio in favor of research.

There has not been much concern about the mix of how funding is distributed up to now but it may become a concern as DSEN approaches full capacity (in terms of responding to Queries). A few interviewees suggested that there may be too much emphasis on capacity development.^{Footnote 23}

As well, interviewees suggested that there may not be sufficient funds to meet the expectations of all stakeholders. While the current level of funding is adequate for the current level of prioritized Queries, it may not be sufficient to support long term demand if the number of Queries submitted increases. The first few years of the DSEN program saw a surge of Queries submitted, due to submitters developing a list of possible Queries in anticipation of DSEN’s launch, but the more recent years have seen a slight decline in the overall number of both submitted and prioritized Queries, with a decrease in the Query rejection rate. Furthermore, there are some Queries which have multiple responses. Ongoing performance measurement and future evaluation will be better able to determine the steady state of the annual Query submission rate; thereby, allowing for improved tracking of changes in the numbers of Queries being submitted and prioritized as well as allocation of resources required to meet program delivery requirements.

3.4.4 Autonomy, independence, and the location of DSEN

The understanding and definition of what independence means varies from stakeholder to stakeholder. Testimony provided to the Senate Standing Committee on Social Affairs, Science and Technology, which held hearings on pharmaceuticals over 2012 and 2013, suggests that some stakeholders are concerned that the pharmaceutical industry may have an influence on CIHR policy decisions due to their partnerships with the national research funding agency. One witness testified that he had several misgivings/reservations about DSEN’s degree of independence from industry (GoC, 2012a). This witness expressed concerns that CIHR’s increasing collaboration and partnering with industry may adversely affect the independence of DSEN. On the other hand, a few witnesses from industry suggested that they were uninformed of DSEN’s research until the network releases its findings. They expressed significant concern that they had no influence in the monitoring of post-market drugs by DSEN. The Senate’s report on the testimony they received stated that industry representatives expressed a desire, at a minimum, to at least be informed when one of their drugs is under investigation (most witnesses were satisfied with the current practice of not informing manufacturers) (GoC, 2013a, 15).

There was some disagreement among interviewees on the appropriate level of independence of DSEN funded researchers in terms of the involvement of stakeholders in defining specific research protocols to respond to submitted Queries. Some interviewees expressed a desire for greater involvement in the development of the objectives and methodology used for researching prioritized Queries. While, on the other hand, there was some concern among other interviewees that decision makers may not have enough technical knowledge to be engaged in the research process in a detailed manner. Additionally, one of the evaluation’s interviewees suggested that DSEN may require increased resources in the future (to address increasing numbers and potentially, more complex, prioritized Queries) and that it could find partnership funding from industry since they felt that there was a sufficiently robust firewall between partnered industry funding contributions and decisions relating to how the contributions are utilized. While there may be divergent views about the independence of DSEN from the influence of industry and the autonomy of DSEN within CIHR, there seems to be sufficient evidence that this concern may be unsubstantiated.

Related to the issue of independence is the location of DSEN within CIHR. The selection of CIHR as a host was arrived at through a systematic process where it was determined that locating DSEN in CIHR was more compatible than the alternatives because it enabled DSEN to take advantage of CIHR’s existing relationship with researchers working in post market drug research, expertise in supporting health-related research, and ability to support the highest quality of research to ensure that the DSEN-supported findings would be seen as rigorous. Evidence collected suggests that the majority of key informants interviewed for this evaluation did not have an issue with DSEN being located in CIHR. Only four of the key informant interviewees discussed this issue with differing opinions about the locating of DSEN in CIHR, with their responses equally falling on a continuum that ranged from it being “critically important” that DSEN remain in CIHR to the opposite extreme where one interviewee opined that it may be better to have DSEN as a separate entity in and of itself.

While the one interviewee’s comments on their support for the continued location of DSEN in CIHR was brief and stressed the need for DSEN to remain part of CIHR, the other three comments explained their positions in greater detail. One interviewee suggested that some stakeholders, during the development of DSEN, did not fully understand the procedures and processes in place at CIHR which affect the timeliness and responsiveness of the network which resulted in early challenges. Another interviewee posited that one way to increase the timeliness and responsiveness of the network would be to have CIHR continue to fund DSEN research, but to have DSEN established as a separate entity which would result in greater independence. Finally, one interviewee suggested a new and independent organization be set up which would be free of any limitations caused by CIHR’s structure and the restrictions around how it provides funding to support research. The interviewee posited that this would result in a more timely and responsive body which would allow it to enter into research contracts that could potentially increase the timeliness of the delivery of research results.^{Footnote 24}

Therefore, while the opinions and arguments of the two interviewees who discussed the relative merits of moving DSEN out of CIHR are relevant, they do not appear sufficiently strong to necessitate a reconsideration of the host analysis conducted. Furthermore, the Canadian Senate has re-affirmed the decision to locate DSEN within CIHR. The recently completed Standing Senate Committee on Social Affairs, Science and Technology’s examination and review of prescription pharmaceuticals in Canada announced, after hearing lengthy testimony from various experts, that it felt that DSEN should remain a part of CIHR since, the opinion of the members of the Senate committee, CIHR is a trusted organization in Canada and that there were no pressing need to re-locate DSEN at the time (GoC, 2013A, 19).

3.4.5 Provincial and Territorial Engagement

The DSEN CO has identified the need to increase provincial and territorial (P/T) engagement in the query process in its November 2012 Dashboard presented to the DSEN SC. Some interviewees suggested that the difficulty in engaging provincial decision makers results from the fact that the capacity to meaningfully engage with the program varies significantly across provinces and territories.

Another factor that may contribute to this challenge is that the provinces, while integrally involved in the work leading towards the establishment of a national PMDSE network, had limited involvement in developing the implementation plan and timeline for DSEN’s establishment, which was initially led by HC. Furthermore, one interviewee also posited that an additional cause could be that provincial and territorial decision makers are less familiar with the DSEN query process.

While there is some evidence that provincial engagement is a challenge, the DSEN CO (and CIHR) has taken actions to better engage them. The Network Advisory Committee (NAC), established after the TBS submission, provided advice for DSEN’s early implementation and engaged regional, provincial and international stakeholders. In addition to more actively engaging provincial and territorial decision makers, the DSEN CO has adopted a number of strategies, including: creating guidance documents that describe how to submit quality Queries^{Footnote 25}; including provincial and territorial decision makers in the development of the MDCA query prioritization tool; and, establishing links with the CADTH to increase awareness among provincial decision-makers (e.g., CADTH has included DSEN as a point of discussion on the agenda for some of its meetings with the provinces).

4. Program Performance

4.1 Key Findings

• Evidence indicates that there has been progress towards achieving many of the program’s immediate and intermediate outcomes.
• The DSEN program has accepted all suitable queries and produced new knowledge which has increased the evidence available to decision makers.
• At the time of this evaluation, seven Queries had been responded to with research results, with three of the responses directly contributing to policy development and helping to inform considerations related to a Common Drug Review recommendation.
• Most interviewees indicated that the establishment of the DSEN program had contributed to greater coordination and collaboration among PMDSE research activities in Canada.
• Query submitters and decision makers perceived a lack of timeliness due to an inability of DSEN research teams to meet the expected timelines to respond to Queries. This poses a key barrier for the program because timeliness is critical to achieve one of its primary goals “to increase available new evidence on post-market drug safety and effectiveness to inform drug regulation, public reimbursement and optimal prescribing and use of drugs.”

4.2 Progress Towards the Achievement of Immediate Outcomes

4.2.1 New knowledge of drug safety and effectiveness has been generated

The review of grant recipients’ progress and final reports that CIHR has received to date reveals that the knowledge creation stemming from DSEN funded projects, as measured by journal articles and other dissemination activities, has been substantial (see Table 4.1). Eight of the ten projects related to Queries from the 2009 initial cohort reported publishing at least one article (the remaining two reported submitting articles). Overall, there were a total of 161 presentations conducted on these 17 projects over the span of three years, with each grant recipient reporting at least one presentation (this could include poster or oral presentations at symposiums, conferences, workshops, etc.). One-third (33.5%) or 54 of these presentations were given in countries other than Canada, including: the United States of America, Mexico, the United Kingdom, Spain, Germany, Switzerland, France, China, Australia and Singapore. This productivity is on par with the performance of other CIHR-supported researchers, as reported in collected end-of-grant reports.

While the publication of journal articles is a proxy that is used to measure knowledge creation, it does not indicate whether or not the knowledge is useful and is making an impact on decision-makers. One way to measure whether the knowledge generated by a specific project is having an impact on others is to look at whether it has been cited in other articles, using citation analyses. The assumption made here is that the more an article is cited, the more it is influencing others, be it positively or negatively. Therefore, the raw citation scores of the articles listed by respondents were collected, using Google Scholar, as an indicator of whether the listed articles were having an impact on the academic community.^{Footnote 26} The analysis of Google Scholar data suggests that the knowledge generated by DSEN-supported research is having an impact on the broader academic community (see Table 4.1).

It is important to note that the number of citations an article receives is affected by how long it has been publically available. For example, an article that was published in 2011 has had 2.5 years to gain citations while an article published in 2013 has had a maximum of six months to gain citations. This, as well as the merit of the article/usefulness of the information it reports are the two reasons why there is great variation in the citation counts (as indicated by the large standard deviation values). The raw citation count ranges from zero citations to 40, with zero, one and four being the most frequent number of citations received.

Note that some Queries are submitted to DSEN by decision makers in order to confirm internal analysis, or to validate synthesis of research findings. Therefore, not all research outputs from DSEN will be of interest for journal editors, so in addition to the articles captured by RRS reports, other content will rest unpublished and must be disseminated directly to relevant stakeholders.

Furthermore, it is important to note that in the particular context of DSEN’s mandate and objectives, these commonly employed measures do not directly map the utility of or impact of DSEN research outputs to those submitting Queries or health care decision-makers more broadly. The utility of the research outputs rests on the specificity of the research result in responding to the central uncertainty underlying any query submitted, which can only be captured by qualitative measures of the submitter’s satisfaction with the research product.

Finally, it should also be noted that much of this data is derived from the initial years of DSEN’s operations whereby a portion of research was conducted in response to open calls to the research community to test the success of researchers proposing research projects to highly targeted questions from decision makers. These open calls did not place requirements on researchers to provide their research outputs directly to the submitter of each research question, which later became a condition of funding for the research teams which comprise the DSEN core methodological capacity. Although the DSEN program partners have worked to identify the research outputs from these early open grants, which are directly relevant to the Queries submitted, not all knowledge produced by these early responses were shared directly with decision makers. Rather, most of this knowledge was disseminated through traditional academic mediums. Furthermore, these data represents knowledge generation and dissemination to a very broad audience and, as such, it is only partially representative of new knowledge provided to the submitters of DSEN Queries.

4.3 Responsiveness of Network and Timeliness of Query Responses

As of March 31, 2013, the DSEN program has received a total of 53 Queries (with the majority of the Queries originating from HC).^{Footnote 27} Of the total number of Queries received, 36 Queries (or 68%) have been prioritized with the remaining Queries either under development (six or 11%) or out of scope/misaligned with the network’s capacity to conduct the required type of research (11 or 21%). Of the 36 prioritized Queries, 22 are under research, seven are completed and seven are prioritized with proposals for funding under review.

Most key informants interviewed agreed that the delivery of DSEN funded research findings have not met the expectations of decision makers with respect to timeliness to date. Interviewees, primarily decision makers, cited several factors that may explain the timelines, including:

• New organizations (such as DSEN) require time to learn how to function efficiently; and,
• Policies and procedures at CIHR, while contributing to the funding of high quality research, can result in attenuated timeframes for the onset of research.

It is important to note that there may be differing interpretations of what timeliness means. The first DSEN Funding Opportunity’s (FO) call for proposals cautioned that any proposed research would need to be completed within a short time frame; however, this was defined differently across grant type. For example, the 2009 catalyst grants supported one-year projects whereas the Team grants were defined as demonstration projects to be completed within a two year timeframe.

While there may be some variation in defining timeliness, there are indications that the DSEN CO is taking actions to make the query response process more time efficient. The DSEN CO has taken steps to reduce the time between query submission and the onset of research. It has established a Scientific Advisory Committee which reviews newly submitted Queries to ensure that they are feasible and specific enough to be responded to by the network teams. As well, the DSEN CO has streamlined the application submission process for securing rapid funding grants. While there is no way to confirm that these actions are solely responsible for reducing the time between query submission and the onset of research, an analysis of data indicates that the gap has decreased substantially (Figure 4.1). The average gap has dropped from a high of approximately 24 months in 2009, the first year of operation of DSEN, to approximately five months in 2013.

Figure 4.1: Number of months between query submission and onset of research by submission year^{Footnote xiii,Footnote xiv}

Long description: Number of months between query submission and onset of research by submission year

Modifications to the current performance measurement indicators are necessary in order to collect the required data that can better determine if the program is resulting in the timely delivery of PMDSE information to query submitters.

It is also important to recognize that early in the implementation of DSEN there was a need to develop lines of communication between decision makers and the DSEN-supported researchers. These parties have spent much time discussing the decision-makers’ needs and expectations in relation to their submitted Queries. Although this creates some delay in the onset of examining the Query due to the level of engagement needed, this is a value added activity, which over time has been becoming increasingly efficient.

The following case study of a query pertaining to Isotretinoin illustrates the importance of collaboration between HC and DSEN in refining a research query.

Isotretinoin is a medication used mostly for severe acne. While highly effective, it has significant documented adverse effects, the best-known and most dangerous being birth defects due to in utero exposure. There are measures in place to decrease the risk of pregnancy in women using Isotretinoin. However, Isotretinoin-exposed pregnancies are still reported internationally. The purpose of HC’s research proposal was to determine more accurately how often pregnancy occurs among Canadian women using Isotretinoin.

HeaIth Canada submitted this query in August 2010. It was the last of 23 topics submitted in the first two years of DSEN’s roll out, received five months prior to the identification of the first DSEN funded team – CNODES. CNODES subsequently identified the research question as one of six potentially actionable Queries to be undertaken in its start-up phase. In early 2012, CNODES contacted HC to discuss possible refinements to the research question. In summer, 2012, following correspondence between DSEN, CNODES and HC, CNODES indicated that it could take on the query as redefined by HC. Most recently CNODES indicated that it had begun to work with various provinces to assess feasibility and limitations around data collection.

Some of the lessons learned from this query include the following:

• Query clarification is a process and a negotiation between the researchers and the decision makers. Each has to understand the other’s perspectives and needs to develop consensus on the way forward with a study. Decision makers feel that they need to be involved in clearly defining the study objectives before research proceeds. This involves considering existing evidence, whether results of the proposed new study would address important knowledge gaps, and whether the new study results plausibly could suggest a particular course of action.
• The establishment of provincial pregnancy cohorts, with harmonized data, for this project has established a data source that CNODES can use to “answer multiple Queries on medications in pregnancy” which was, to the researcher; a tangible reward for their work on the Isotretinoin query.

While the researchers have not completed their research at the time of data collection, they have identified unintended benefits stemming from the work and DSEN as a whole. The DSEN network has allowed researchers to better access medical data that enables them to undertake complex studies that had not been feasible prior to the establishment of DSEN. DSEN therefore provides a way for HC and other P/T health authorities to obtain answers to important drug safety questions that it has identified, answers which may not otherwise be obtainable at the time of decision-making.

4.4 DSEN Responsiveness and Delivery of Evidence – Suitability of methods and mediums

The RRS and qualitative data collected by the evaluation identify four primary methods used to share DSEN research results with stakeholders: reports; direct communication between researchers and decision makers/query generators; academic publishing; and, presentations.

To date, most of the information shared with present query generators (i.e., HC) has taken the form of academic papers which are highly technical in nature. The current format of the responses to Queries has not challenged the capacity of the query submitters. For example, one interviewee noted:

Right now the majority of the research coming back is in the format of a scientific journal paper… which is not particularly problematic for people working in the decision-making realm who have that same level of scientific training.

While this format suits the needs of the primary query submitter, it may not be suitable for other decision makers such as representatives from the provinces or the Canadian Agency for Drugs and Technologies in Health. Further, future decision makers may require different response formats to meet their needs and capacity.

Additionally, the current media/fora used to convey research findings are not necessarily suitable for broader dissemination, to other stakeholder groups. As one interviewee suggested, one of the deliverables that should be produced by researchers is “a lay person understandable press release”. The DSEN CO is currently developing a knowledge translation strategy that includes plain language findings documents that can be posted on DSEN’s website to broaden the dissemination of their network’s research results beyond the query submitter and improve transparency for the program. Furthermore, some interviewees expressed a desire for all DSEN SC members to receive query responses, regardless of whether or not their constituency submitted the initial query.

The following is an example of one of DSEN’s research teams, CNODES, being proactive in their dissemination of findings beyond the query submitter. CNODES developed a publication announcement for an article they produced on the results of a DSEN-supported study examining statins and published by the British Medical Journal, one of the world’s most prestigious medical journals. News of their results was published in such places as the UK’s Daily Mail, Heartwire; Pharmacology Weekly; Forbes; CBC; and Pulse.^{Footnote 28}

Another example of a proactive dissemination strategy implemented by DSEN-supported researchers is an examination of the use of second-generation anti-psychotics by children (usually in treatment of Attention Deficit Hyperactivity Disorder). The researchers found that “there is good evidence that second generation antipsychotics cause both metabolic and extrapyramidal side effects in children. This evidence comes from randomized controlled trials of these medications in children, and prospective cohort studies”.^{Footnote 29} Besides providing the query submitter with a response, the research team has published several journal articles and has developed several drug specific safety-related clinical guidelines which have been made available to the public on a publically accessible website.^{Footnote 30} Each guideline provides drug-specific recommendations for monitoring drug safety in children, since “the risk and timing of side-effects vary by medication”.^{Footnote 29}

Findings from key informant interviews also suggest that no single response format or template will meet the needs of all decision makers/query generators. In light of this, some interviewees suggest that increased dialogue is necessary between researchers and the query generators to better ensure that the response to each query is provided in the appropriate medium, format and has the necessary content to meet the needs a specific decision makers/query generator.

4.5 Use of DSEN-generated Evidence to Inform Decisions

As of March 31, 2013, the DSEN research teams had delivered responses to seven Queries. There was only a limited amount of evidence collected through key informant interviews and document reviews relating to how this information provided evidence for the decision-making of query submitters. Available evidence suggests that DSEN query responses have had a limited impact on decision making. Members of the Executive Working Group and SC members from HC identified a few areas where DSEN-generated evidence had an impact on HC decisions, policies and practices. Two of the examples provided by interviewees were:

• The evidence generated in response to a query confirmed a risk decision and assisted with making recommendations on final labeling on second generation antipsychotics and cardio-metabolic effects in children and adolescents. According to one interviewee, the query “stimulated a review of how many First Nations and Inuit children were receiving antipsychotics and whether that number was increasing, so I would say that that particular query had the most impact.”
• The evidence provided in response to another query related to drugs prescribed to children and pregnant and nursing women, according to one interviewee, may have informed policy direction on “the most prescribed and highest risk drugs [by] looking at differences internationally in prescribing practices.”

Findings from the case study of the response to a query regarding the comparative effectiveness of anticoagulants, used to reduce the risk of thrombosis in at-risk patients, indicates that DSEN-generated evidence contributed to CADTH’s decision to issue a recommendation, through its Common Drug Review for the continued use of the standard anticoagulant. The query was submitted by a provincial drug plan manager on August 31, 2011, and focused on the comparative effectiveness of new oral anticoagulants (NOACs), such as Dabigatran, Apixaban and Rivaroxaban to Warfarin which is the standard anticoagulant and has been used for over 50 years. These drugs are used to reduce the risk of thrombosis in at-risk patients. There were several reasons for the submission of the query including:

• The rapid uptake of NOACs and which has raised questions among provincial drug plan managers regarding drug coverage and safety;
• A trial called RELY in Japan suggested safety concerns related to Dabigatran; and,
• There were no known cost effectiveness comparative studies of NOACs to Warfarin.

One of DSEN’s seven research teams, the Canadian Collaboration for Drug Safety, Effectiveness and Network Meta-Analysis, submitted a proposal and were funded to conduct a comparative study of the cost-effectiveness and safety questions of NOACs in comparison to the standard - Warfarin. The Query response was completed and submitted to CADTH in April 2012; CADTH participated in the study since it felt that the research was of interest to one or more P/T jurisdictions. Warfarin was found to be just as effective as other anticoagulants at a much lower cost (a dose of Warfarin is approximately 1/3 the cost of a similar dose of Dabigatran). The research team disseminated the results of their study through a detailed report available on CADTH’s website (Wells, 2012), through presentations and through CADTH’s one-page summary of the study’s key findings.^{Footnote 31}

The results contributed evidence informing a Common Drug Review recommendation, in September 2013, on the continued use of Warfarin. The study’s results, through the recommendation, are informing provincial drug plans in their negotiations with respect to the price of NOACs. As well, the findings have been translated into specific clinical recommendations targeted towards prescribers. Furthermore, the project resulted in capacity development by directly involving several PhDs and Master’s students; one of whom is exploring knowledge dissemination techniques to make the results of network meta-analysis and economic evaluations more understandable.

It is also important to note that the limited evidence of contribution towards or influence on policy for all seven submitted responses does not mean that the responses have not had any influence. For example, evidence submitted by DSEN researchers could result in no modification or impact on policy if their findings suggest there is no cause for concern in relation to the topic of the submitted query; however, the decision to take no action is still a policy decision that was based, in part, on evidence submitted by DSEN. Furthermore, it is important to note that the DSEN program is relatively new and the research to respond to many of the submitted Queries was initiated by DSEN teams in 2011 and later. In addition to the relatively recent launch of DSEN program, is the time required for findings submitted in response to a query to influence decision-making at a query generator’s organization. As one interviewee noted:

“[Findings have a] trickle-down effect in terms of policy changes… is sometimes a bit slow so first of all, the users will have to think about the data to decide what kind of regulatory changes might be necessary so I would guess it would probably be a year anyway till we see major regulatory changes on the basis of this data.”

The time between the submission of the query results for the NOAC study to CADTH and the issuing of the recommendation seems to support this statement.

4.6 Coordination, Collaboration and Linkages to Address PMDSE Evidence Gaps

DSEN supported research has been conducted by researchers working in collaborations with each other. The ten completed catalyst grants, issued before the establishment of the seven methodological teams, had an average of approximately four co-applicants/principal investigators. In 2011, the DSEN program distributed multi-year grants to support seven teams of researchers distributed across Canada that focus on five specific methodologies, (see Figure 4.2). Furthermore, DSEN researchers collaborate with researchers from other countries (such as Greece, the United Kingdom and the United States of America). The seven teams are comprised of approximately 170 researchers.

Figure 4.2: Geographic Distribution of the DSEN Teams’ Researchers

Long description: Geographic Distribution of the DSEN Teams’ Researchers

The DSEN program has also taken proactive steps to expand its network, by conducting numerous networking and engagement activities, both nationally and internationally (see Table 4.2). The DSEN has also engaged stakeholders from HC, the Provinces and Territories, health technology assessors (CADTH and L'Institut national d'excellence en santé et en services sociaux), and international organizations (such as bilateral meetings with the US Food and Drug Administration and the European Medicines Agency, and the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance and the USFDA Mini-Sentinel program).

All key informant interview participants agreed that DSEN has made an important contribution towards increasing communication between decision-makers, CIHR, HC and PMDSE researchers. Interviewees reported that DSEN has created a national forum/body in PMDSE that has brought greater coordination to these previously fragmented bodies/groups. The key achievements of DSEN identified by interviewees include: better leveraging of expertise across various groups of PMDSE researchers; developing PMDSE capacity in Canada; and, fostering culture change by encouraging its researchers to be more willing to share preliminary results with decision-makers.

In addition, DSEN has succeeded in better coordinating and facilitating access to drug databases. To date, DSEN has secured access to the drug use records of over 40 million individuals for its research teams through access to data captured by provincial drug plans in Canada, the United Kingdom’s Clinical Practice Research Datalink (CPRD) and the United States’ MarketScan Database. For example, one query examining Isotretinoin that is nearing completion relies on data from several registries from multiple provinces. The lead investigator suggested that the use of these collated data would not have been possible in the absence of DSEN.

To date, DSEN has established a limited number of linkages and partnerships with other national and international organizations in the domain of PMDSE research. Most interviewees acknowledged that only a few years has passed since DSEN’s launch and that it takes time to establish solid partnerships and linkages, but note that there has been some progress in this area. For example, the DSEN CO is in the process of developing a collaboration agreement with CADTH to support P/T decision makers by identifying and refining jurisdictional evidence needs, producing research (clinical and economic) and disseminating findings to the broader decision making community. Specifically, DSEN and CADTH will collaborate on a joint project to:

• Coordinate jurisdictional input to DSEN on real world drug safety and effectiveness topics of relevance to Provinces and Territories,
• Produce evidence to support jurisdictional research needs of post-market drug safety and effectiveness,
• Support dissemination of the DSEN evidence related to these topics through Knowledge Mobilization efforts to Provinces and Territories.

The intent of this partnership is to have CADTH act as a point of contact for the provinces and territories. It is expected that having CADTH as a central query submitter will facilitate P/T engagement, allowing them to rely on one entity with relevant expertise that can work with them to develop appropriate query submissions. Using CADTH to represent provincial issues will likely reduce the challenge DSEN currently has in engaging the provinces individually, given that each province is autonomous and has its own policies, priorities and infrastructure. Findings from the case study of the response to the NOAC query provide a good example of the synergies that can be realized by having CADTH act as a point of contact for federal/provincial/territorial drug plans PMDSE questions and Queries.

4.7 Direct and Indirect Capacity Development

Although the evaluation focused on the program’s outputs and immediate outcomes, data was collected that indicated that there has been some progress towards achieving the intermediate outcome of increased capacity in Canada to undertake high-quality post-market research. DSEN supports capacity development directly through awards for individual researchers, and indirectly, by providing funding for research projects that develop capacity through the involvement of students, trainees and other researchers/stakeholders. Capacity is further developed by negotiating access to additional data sources (which increases the network’s ability to undertake a variety of analyses) and by developing new methodologies which can then be applied to Queries that were previously unanswerable.

As outlined in Table 4.3, DSEN has invested approximately $4 million in direct support of capacity development. Recently, the network has invested $1.8 million over five years in the career development of six new investigators, starting in 2012-13. Furthermore, DSEN assumed responsibility for funding the Drug Safety and Effectiveness Cross-Disciplinary Training (DSECT) program in 2012, which supported 28 fellows in drug safety and effectiveness research in 2012.^{Footnote 32} In addition, DSEN has further supported capacity development by providing opportunities for trainees to participate in conferences, promoting trainees and new investigators participation in network events and by supporting the development of a national curriculum for training new drug safety and effectiveness researchers.

The seven DSEN research teams also have a responsibility to provide indirect capacity development within their team funding envelope through funding research projects involving, and providing training opportunities to students, trainees and researchers.

Given the focus of the evaluation on the immediate outcomes and the information available via the Research Reporting System (RRS) reports submitted, there is limited evidence to assess indirect capacity development that may have occurred. In the case of RRS reports, the limited information regarding capacity development is due to a reduction in the amount of information recipients of smaller grants, which include many of DSEN’s early grants, are required to report on in the end of grant RRS report.^{Footnote 33} A few RRS annual research team reports however did include some brief information on indirect capacity development in the narrative portion of the report. For example, the Canadian Network for Observational Studies (CNODES) reported, in their first annual report, that they had 35 trainees involved with the work they have conducted so far. Overall, there are insufficient data to assess the full performance of DSEN’s indirect capacity development initiatives to date.

It is important to note that DSEN also interprets capacity development as increasing data access for its researchers (thus improving their capacity to conduct larger studies). An example of DSEN’s success in this area was the establishment of a harmonized database of provincial pregnancy cohorts which has the potential to increase the speed with which DSEN’s research teams can respond to Queries that involve/target pregnant women (see Isotretinoin case study in Section 4.3). Furthermore, DSEN also includes methodological advancements in its definition of capacity development. Seven papers related to methodological advancement have been published by its teams since 2011, with four having between 7-12 citations each (as reported by Google Scholar). This is further evidence that DSEN is showing progress towards achieving this intermediate outcome.

5. Relevance

5.1 Key Findings

• Evidence from the evaluation indicates that:
  • DSEN program responds to a continued need.
  • DSEN is consistent with federal government roles and responsibilities.
  • DSEN aligns with federal government, HC, and CIHR priorities.

5.2 Continued Need

The evaluation evidence indicates there is a continued and likely increasing need for active surveillance to ensure the safety and effectiveness of marketed drugs. The DSEN program works to address this need by coordinating, generating and promoting the use of PMDSE research to inform F/P/T decision-making. Active post-market surveillance mechanisms, such as DSEN, are important for patient safety because decision makers want to ensure that the most effective pharmaceuticals for the safe treatment of illnesses are used in order to support the cost-effective, evidence-informed operation of medical drug plans.

The Canadian Institute of Health Information has estimated that Canadians spent $29.3 billion on prescription drugs in 2013 (CIHI, 2014), which represented 13.9% of all health expenditure for that year. Furthermore, the recent Canadian Rx Atlas, 3rd Edition (Morgan et al, 2013) re-affirms the importance of pharmaceuticals. It reports that three out four Canadians fill out at least one prescription every year. The report also predicts that the costs associated with prescription drugs in Canada will sharply increase with the introduction of increasingly specialized, and expensive, drugs to treat specific ailments. In this context, there is a need to identify the most efficacious drugs to enable P/T drug planners to keep their programs as efficient as possible while ensuring that Canadians can access the safest, most effective pharmaceuticals. Findings from the case study of the response to the Dabigatran Query provide an example of how DSEN is addressing the need for evidence to inform drug plan management by providing evidence that second generation anticoagulants are not necessarily more efficacious than Warfarin, which has the potential for considerable cost-saving for P/T drug plans given that the cost per dose of Warfarin is one-third that of second generation anticoagulants.

Findings from key informant interviews indicate that DSEN is needed, with several interviewees suggesting that it would be detrimental to return to how PMDSE research was conducted prior to the launch of DSEN. According to interviewees, the loss of DSEN would result in fragmented and uncoordinated research which could reduce the availability of credible research for decision makers meaning they would have to make decisions based on the resources and information that are available to them. While some PMDSE research would likely continue, it would be researcher driven and less likely to address the information needs of decision-makers since the research would not be focused to address questions (queries) posed by decision makers.

The increased creation and use of PMDSE knowledge is an important mechanism to reduce the number of Adverse Drug Reactions (ADRs) and their associated harms and healthcare costs. Adverse Drug Reactions (ADRs) are rare, serious and unexpected risks of a drug when used as authorized for the market or in cases where health care professionals make treatment decisions and prescribe pharmaceuticals “off-label” (i.e., outside the manufacturers identified or authorized use for the approved drug).

In most cases, ADRs are voluntarily reported when an adverse reaction to a drug/medical treatment is observed (only manufacturers are required to complete ADR reports) — it is estimated that ADR reports only capture between one percent and ten percent of ADRs (Wiktorowicz et al. 2010; Hazell & Shakir, 2006). The failure of passive ADR reporting to fully capture adverse drug reactions has led to experts in post market drug safety calling for “active” surveillance (see Adler Group, 2006; Health Council of Canada, 2010) to provide early detection of safety problems and prevent greater public harm. In addition, the medical healthcare costs associated with the treatment of ADRs are significant, for example, the estimated medical healthcare costs associated with the treatment of ADRs among seniors (66 years of age and older) was $35.7 million in 2007. DSEN’s active surveillance through its PMDSE research on drugs identified by F/P/T decision makers helps to reduce the uncertainty about drug response, allow for better risk-benefit profiling for decision makers, and identify dangerous issues before large number of users are affected.

The recent Standing Senate Committee on Social Affairs, Science and Technology (SOCI) report on pharmaceutical drugs, Prescription Pharmaceuticals in Canada: Post-Approval Monitoring of Safety and Effectiveness (GoC, 2013b), and the Government of Canada’s response to the report, suggest that PMDSE evidence is still needed by F/P/T decision makers and that DSEN is well situated to provide it. Specifically, the SOCI report recommends the expansion of DSEN’s mandate, citing a need for PMDSE data on prescription drug effects on sub-groups of Canada’s population (e.g., pregnant women, children, Aboriginal people) and indicates that a formal mechanism be established whereupon DSEN can ensure that its recommendations (based on the findings of research conducted by its methodological teams) are translated into action by relevant regulatory bodies; and that the program be made permanent.

5.3 Consistency with Federal Government Roles and Responsibilities

The evaluation findings show that the DSEN program’s role to fund and facilitate PMDSE research is consistent with the roles and responsibilities of the Government of Canada.

The DSEN program closely aligns with CIHR’s mandate, as per the Canadian Institutes of Health Research Act, is “to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system" (Bill C-13, April 13, 2000). DSEN directly contributes to the achievement of this mandate by facilitating the creation, dissemination and use of health-related knowledge, as well as the development and maintenance of Canadian health research capacity, within the area of PMDSE. In addition, the aforementioned SOCI report (GoC, 2013a) acknowledges the federal government’s role in supporting PMDSE research and recommends that DSEN be established as a permanent entity and have on-going and sustained funding which is consistent with a 2010 report of the Health Council of Canada which stated: “The federal government needs to commit to secure, stable and ongoing funding in order for the DSEN to be able to adequately plan and carry out long-term research” (2010, p. 42). The government’s response to the Senate report agreed, in principle, with the committee’s findings. Key informant interviews and case studies also corroborate these findings.

5.4 Consistency with Government and Agency Priorities

The FCSAP has three action areas: Active Prevention; Targeted Oversight; and, Rapid Response. The DSEN program is one key element which assists in ensuring that FCSAP’s goal of achieving “Targeted Oversight” for health products. DSEN does this by generating new evidence to support the ongoing assessment of the risk-benefit profile of drugs after they enter the market and by building capacity. Health Canada’s recent Report on Plans and Priorities (RPP) for 2014-15 affirms their regulatory role in the governance of the safety of products including food, pharmaceuticals, medical devices, natural health products, consumer products, chemicals, radiation emitting devices, cosmetics and pesticides.^{Footnote 34} Therefore, DSEN’s strategic direction continues to support HC’s priorities.

In addition to the HC priorities, the goals of the DSEN program support and contribute, either directly or indirectly, to two of the three advantages outlined in the federal government’s Mobilizing Science and Technology to Canada’s Advantage (S&T Strategy) (Industry Canada, 2007 & 2009) — Knowledge Advantage and People Advantage. The program has contributed to the Knowledge Advantage, which identified basic and applied research in many areas (including health) as a sub-priority, and has helped maintain “Canada’s international reputation for research excellence” (Industry Canada, 2009) through the creation of new health-related knowledge about the safety and effectiveness of drugs in Canada. Its unique and innovative approach has garnered worldwide interest, as evidenced by the numerous countries where DSEN-supported researchers have made invited presentations on their research and by the international news coverage of some of the findings of DSEN research teams.

In terms of the People Advantage, DSEN’s capacity development activities have helped “develop and maintain Canada’s health research capacity” and has contributed to the building of “the best-educated, best-trained and most flexible workforce in the world” (Industry Canada, 2009). For example, DSEN supported research and funding opportunities have directly supported over 170 PMDSE researchers through team grants, funded an additional seven new investigators through awards and bridging grants, and indirectly developed capacity by training graduate and post-graduate students through their participation in the research teams.

The evaluation findings indicate that the DSEN program has contributed to the achievement of CIHR’s goals outlined in its strategic plan Health Research Roadmap: Creating innovative research for better health and health care. The Roadmap outlines a set of activities linked to four strategic directions:

• To invest in world-class research excellence;
• To address health and health system research priorities;
• To accelerate the capture of health and economic benefits of health research; and,
• To achieve organizational excellence, foster ethics and demonstrate impact.

DSEN’s investment in PMDSE capacity development contributes towards maintaining a sustainable world-class research enterprise. Its support of applied research into PMDSE contributes and accelerates the capture of health and economic benefits, for example, the use of DSEN generated PMDSE evidence will likely lead to increased efficacy and efficiencies in the management of F/P/T drug plans. Finally, the continued improvement in the mechanisms used by the DSEN CO to support PMDSE research in Canada contributes towards the CIHR’s overall organizational excellence.

6. Conclusion and Recommendations

6.1 Conclusions

The evaluation found that the DSEN CO has established Canada’s first national PMDSE network within the planned timelines, supported the development of new PMDSE knowledge and capacity building in Canada, and has made considerable efforts at developing and establishing management and performance protocols and procedures to enable DSEN to achieve its immediate and intermediate outcomes.

The establishment of DSEN has resulted in the creation of a national PMDSE forum and has brought greater coordination to PMDSE-related research activities. According to interviewees, this has been achieved through better leveraging of expertise across various groups of researchers, developing PMDSE capacity in Canada, and fostering culture change by encouraging an increased willingness among researchers to share preliminary results with decision-makers. There is, however, evidence that some primary stakeholders are not yet entirely clear on the role and function of DSEN and their relevant roles and responsibilities within the program. Specifically, there are three aspects of the program which require clarification: the roles and responsibilities of the DSEN Steering Committee; the time taken to prioritize and respond to queries; and the level of independence of DSEN’s researchers.

During the period under review, the DSEN CO established seven methodological-specific research teams and received a total of 53 Queries of which 36 have been prioritized for research during the period under review. Of the 36 prioritized Queries, research is complete for seven (12 Queries were completed as of September 2014), is ongoing for 22, and is planned pending resources for the remaining seven Queries. To date, DSEN research has informed a few PMDSE decisions relating to the confirmation of a risk and making recommendations to final labeling for a drug, informing a policy decision and contributing to a Common Drug Review recommendation; however, this should be considered in the context of the relatively recent launch of the DSEN program, the small number of completed Queries and the time required for the response to a Query to influence decision-making. Overall, the DSEN program has distributed a total of 101 grants and awards, with 62 of these being used to support projects related to prioritized Queries and 39 used to support networking, knowledge translation and capacity development.

While most interviewees expressed opinions about the timeliness of the delivery of evidence and the need to track timeliness, there are no data on timeliness that is systematically collected to date. Therefore, there is a need to strengthen the communication and tracking of information processes around the timelines of query submissions and responses. In addition, many of the DSEN grants analyzed in the reporting period were only required to complete abbreviated versions of the RRS report, which do not collect training and capacity development related data. This makes it difficult to estimate the total number of trainees (a key measure of capacity development). As a result, there is a need for the identification and implementation of indicators to better measure the program’s impact in the area of training and capacity development.

The assessment of program efficiency conducted by this evaluation were limited due to the inability to separate out the DSEN component of HC expenditures related to Targeted Oversight and, as a result, had to be estimated, with the exception of the 2012-13 fiscal year. Over the period under review, the estimated total delivery costs of DSEN were 33% ($8,656,676/$25,275,571) of total program expenditure and decreased from 100% in 2008-09 (the first year of operation) to 20% in 2012-13. The DSEN CO and HC related delivery costs were approximately equal over this period: 16% ($4,266,169/$25,909,821) and 17% ($4,390,507/$25,909,821) of program expenditures, respectively. For fiscal year 2012-13, the delivery costs of the DSEN CO were 10.4% ($1,129,949/$10,808,622) of total expenditures and total delivery costs, including both DSEN CO and HC expenditures were 19.5% ($2,116,292/$10,808,622) of total expenditures. Going forward, 2012-13 cost data can be used as a baseline for the program within the Treasury Board allocations for operating and grants and awards expenditures and improvements should be made to the tracking and reporting of program expenditures across program activities areas, such as in capacity building, evidence generation, and knowledge translation.

The DSEN program is working to address the continued need for the active surveillance of drug safety and effectiveness in Canada. Its goals and objectives are consistent with the roles and responsibilities of the federal government and align with Health Canada (FCSAP) and the federal governments’ priorities on targeted oversight of health products, as well as CIHR’s strategic directions.

6.2 Recommendations

The following recommendations are intended to support DSEN in positioning the program to achieve its objective of better informing pharmaceutical post market drug safety and effectiveness decision-making across the Canadian health care system.

1. The DSEN CO, in consultation with its partners and stakeholders, needs to examine key design and delivery features of DSEN to identify areas where efficiency and effectiveness can be enhanced. Key areas for further examination include:
   1. Clarify key aspects of DSEN’s operation with program stakeholders
      The clarification and common understanding of several aspects of DSEN’s operation would help strengthen the program:
      • Effective formats for communicating research results in response to Queries to stakeholders according to their different mandates and information needs (i.e., knowledge translation).
      • The prioritization of submitted Queries.
      • The appropriate level of independence of DSEN researchers in the context of their interactions with query submitters to define research protocols in response to Queries.
   2. Establish service standards for the Query submission and response process to clarify expectations for Query submitters and support performance measurement.
      • Decision makers require clear and transparent timelines for the delivery of research results in response to Queries to make timely decisions on the safety and effectiveness of marketed drugs.
      • There is a need for improved dialogue between Query submitters and DSEN researchers at the outset of submission to clarify the information required to respond to the Query and the context and timelines of the decision-making process the research result will inform or support.
      • The DSEN CO should develop and implement service standards, such as target response times, to enable submitters and research teams to establish common expectations and agreement on timelines and milestones, taking into consideration the methodology and scope of the proposed research, at the outset and throughout the Query submission and response process.
2. The DSEN CO, in consultation with Health Canada and CIHR, should review the current performance measurement strategy to identify changes to better monitor performance against expected outcomes.
   • The DSEN CO should identify the indicators to collect and track information to better monitor, assess and communicate the performance and impact of DSEN. In particular, additional indicators should be developed relating to timeliness, program expenditures, indirect training and capacity development, and the longer term benefits of query responses on the Canadian health care system.
   • In terms of program expenditures, DSEN needs to track and report on expenditures in greater detail in order to: separate out DSEN related expenditures from the broader Targeted Oversight activities of FCSAP; and, map the use of grant funds by research teams to support PMDSE research, capacity development, knowledge translation and network activities.

Appendix A: Evaluation Questions and Methodology

DSEN Evaluation Framework

Evaluation Methodology Description

Consistent with TBS guidance and recognized best practice in evaluation (e.g., McDavid & Hawthorn, 2006), a range of methods will be used to triangulate evaluation findings. The approach of using multiple methodologies involving both quantitative and qualitative evidence is designed to ensure that the evaluation findings are robust and credible and that valid conclusions can be drawn about the performance of the programs.

Document Review

This component will provide background information on the history and objectives of the DSEN program as well as insight into any important shift or change that may have occurred during the program implementation and delivery of services. It will also contribute evidence for answering some of the evaluation questions and issues. Furthermore, the document review will assist with the development and formulation of the interview questions. The following are examples of the relevant documents that will be included in the review:

• Treasury Board Submission, HC Project Charter, FCSAP Logic Model, National Pharmaceutical Strategy
• Departmental documents, such as CIHR and HC Business or Operational Plans, Departmental Performance Reports, Reports on Plans and Priorities, and their Program Activity Architectures/Management Resources and Results Structures;
• Parliamentary and Senate Committee reports
• Relevant Government-wide documentation, including recent Federal Budgets and Speeches from the Throne;
• Communications and other promotional products, website, budgets, reports, meeting minutes, and work plans; and,
• On-going DSEN performance measurement information.

The document review may also include relevant documents related to the design, implementation, impacts and evaluations of similar funding programs to allow an analysis and comparison of similar initiatives in other jurisdictions including those internationally, if available.

Interviews

A total of 17 interviews served as an important source of information for the evaluation by providing information that was used as a line of evidence for all evaluation issues and questions. The interviews were structured to address issues to which the other lines of evidence could not, or had limited ability to, contribute. Only two of the 17 interviewees were from the DSEN CO, with the remaining 15 interviewees belonging to various stakeholder groups. The interviewees were divided into three distinct groups:

• Steering Committee members – five were interviewed;
• Working Group members – four were interviewed; and,
• Partners – eight were interviewed.

Participants who were on the Steering Committee but also sat on the working group were included in the analysis for the working group. There were five participant interviews of members of the Steering Committee. Two individuals were senior policy decision makers and three were academics/researchers. The senior policy makers have held their positions for less than three years while the academics/researchers have been in their roles for over ten years. Two of these individuals cited involvement in the conceptual work that preceded the establishment of the DSEN initiative.

There were four participant interviews of members of the Working Group; two DSEN staff and two staff from Health Canada. Both DSEN staff members had been with the initiative from the outset. One of the Health Canada staff members had over 20 years seniority in Health Canada while the other had four years’ experience. Out of the four Working Group members, two had been involved in the conceptual work that preceded the establishment of the DSEN initiative.

There were eight participant interviewees identified as partners; some of whom also sat on the Steering Committee. Two interviewees represented provincial agencies, one a national agency and five Health Canada. Their experience in post-market research ranged from two to 10 years.

Research Reporting System Performance Measurement Data

The Canadian Institutes of Health Research (CIHR) formally launched the Research Reporting System (RRS) instrument on March 31, 2011. CIHR requires Nominated Principal Investigators (NPIs) to report on their CIHR-supported research results via the RRS. The intention is to use these RRS reports for a variety of internal and external purposes, including to obtain richer evidence on the effectiveness of CIHR funding programs, to advance CIHR's Knowledge Translation mandate, and for contributing evidence towards CIHR’s accountability within the Federal Government and to Canadians for their investment in health research. CIHR also uses the RRS data for a variety of internal uses; including to better manage the process of funding health research.

All NPIs on Open Grants and selected Priority Announcements where the authority to spend funds expires as of March 31, 2011 or later must submit end of grant reports via RRS. NPIs have 18 months after the end of the grant period to complete their reports, and CIHR provides on-going support to assist NPIs in this task.

The RRS tool is accessible through CIHR’s ResearchNet platform which enables the pre-population of certain fields in the online survey with data provided at the grant application stage. The tool is organized into seven distinct sections (each with several sub-sections): (a) Profile and Grant Information; (b) Research and Knowledge Translation Practices; (c) Research Findings; (d) Research Capacity and Training; (e) Advancing Knowledge; (f) Broader Impacts; and (g) Research Context.

Google Scholar Data

Google Scholar was used to collect raw citation data on the approximately forty articles published by DSEN-supported researchers that were related to funding received from the program. Google Scholar has been an alternative, and free, source of citation data since 2004. One of the benefits of Scholar, besides being free, is that it captures a wide array of materials within its analyses including “journals, repositories (RePEc, Arxiv and Social Science Research Network), databases (Cochrane database of Systematic Reviews), conference proceedings (IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Proceedings of the ESSCIRC, Proceedings of SPIE, AIP Conference Proceedings) and working papers (NBER Working Paper Series)” (Cabezas-Clavijo & Delgado-López-Cózar, 2012, p. 3).

The broader inclusion of citation sources by Google Scholar makes comparison of raw citations collected through other databases like Scopus or Web of Science difficult, since these data sources include only a selected number of the most prominent journals (most being English-language journals) (personal communication with Jean-Pierre Robitaille, a bibliometrician at Observatoire des sciences et des technologies, July 29th, 2013). While there may be a divergence in the number of raw citation scores an article may have, the relative ranking of the authors are similar, regardless of the data source. One study (Meho & Yang, 2007) suggested that while the absolute number of citations received by a paper may differ between Google Scholar, Scopus and Web of Science, the ranking of researchers through the use of Google Scholar and the merged total number of citations in Scopus and Web of Science were similar. A second study suggested that the increased number of citations recorded by Google are not error, but likely due to the inclusion of a broader source of data sources (books, thesis, etc.) (van Aalst, Jan, 2010). Furthermore, the differences are not dramatic, since all three citation databases use academic journals as the basis of their calculations (personal communication with Jean-Pierre Robitaille, a bibliometrician at Observatoire des sciences et des technologies, July 29th, 2013). The most serious critiques of Google Scholar data are the difference in search results when using key word searchers (there is greater convergence between Scopus and Web of Science), as well as the increased country ranking results within Google Scholar, which may be caused by Scholar’s greater inclusion of non-English data sources; and, the challenges in normalizing results between information sources (i.e. scientific journal versus conference proceedings, books, reports, etc.). These issues make the evaluation and comparison of journal performance difficult (Cabezas-Clavijo & Delgado-López-Cózar, 2012).

While these limitations are valid, they were less applicable to the analysis included in this evaluation, which was to determine whether or not knowledge produced by DSEN-supported researchers was having an impact on the broader community (as measured by the number of citations DSEN-supported papers received). As well, the evaluation was also interested in citations outside of academic journals. Therefore, data collected through Google Scholar was included in the evaluation of the program, given that “Google Scholar citations can provide an acceptable indicator of impact” of the DSEN-related papers on the broader community (van Aalst, 2010, p. 391).

Case Studies

Two purposefully selected case studies were conducted involving a total of six interviews as well as document review to support the background context of the case studies. One principal investigator was interviewed per case study, with an additional two query-related decision makers.

The case studies were written using a standard template (set of key questions) and informed by key informant interviews, review of query submissions, research reports as well as other information provided by the key informants. Due to time constraints, the case studies were not reviewed by the principal investigator as is standard procedure in developing case studies.

Administrative Data Analysis

The administrative/financial data of CIHR and HC was used to determine the cost of the program.

Crosswalk of TBS Core Evaluation Issues by Relevant Sections of DSEN Evaluation Report

Appendix B: DSEN Logic Model

Long description: DSEN Logic Model

Appendix C: Life Cycle Approach Model

Long description: Life Cycle Approach Model

*Note that this diagram was developed by Health Canada.

Clinical Trial Review

• Clinical Trial Applications
• Registration of Clinical Trials
• Clinical Trial ADR Reporting

Product Submission

• Safety, Efficacy, Quality
• Benefit-Risk Assessment
• Basic Scientific Information
• Results of Clinical Studies
• Product Information: Label, Product Monograph, Package Leaflet
• Risk Management Plan including Pharmacovigilance Plan

Authorizations

• Types
• Terms and Conditions
• Ability to Amend
• Ability to Suspend and Revoke
• Obligations on MAH
  • Reporting
  • Post-Market Studies
  • Risk-Mitigation Measures

Ongoing Reporting

• Submissions for New Indications
• ADR Reporting including PSURs
• Post-Market Studies / Trials
• Benefit-Risk Communications

Re-Evaluation

• Opportunity to Re-Evaluate Benefit-Risk Profile when Necessary
• Safety
• Efficacy
• Utilization
• Use in Special Populations

Appendix D: DSEN Organizational Chart (13–03–2013)

Long description: DSEN Organizational Chart (13–03–2013

Appendix E: DSEN Query Process Map

Long description: DSEN Query Process Map

References

Brookings Institution (2012). Highlights from the Fifth Annual Sentinel Initiative Public Workshop. Accessed 13–03–2013.

Canadian Institute for Health Information (2014). Prescribed Drug Spending in Canada, 2012: A Focus on Public Drug Programs [ PDF (1.4 MB) - external link ]. Accessed 23–04–2014.

Fernandopulle, R. B.; and Weerasuriya, K. (2003). What can consumer adverse drug reaction reporting add to existing health professional-based systems? Focus on the developing world. Drug Safety, 26, 4, 219-225.

Fox, G. and Otten, N (2007). Medicines that Work for Canadians: Business Plan for a Drug Safety and Effectiveness Network. Accessed on 18–12–2012.

Government of Canada (2012a). Proceedings of the Standing Senate Committee on Social Affairs, Science and Technology, Issue 26 - Evidence - November 21, 2012. Accessed on 18–12–2012.

Government of Canada (2013a). Twentieth Report: Prescription Pharmaceuticals in Canada: Post-Approval Monitoring of Safety and Effectiveness. Accessed on 19–07–2013.

Government of Canada (2013b). Twentieth Report: Prescription Pharmaceuticals in Canada: Post-Approval Monitoring of Safety and Effectiveness – Government Response. Accessed on 19–07–2013.

Government of Canada (2012a). Study on prescription pharmaceuticals in Canada - Issue No. 25, November 7 and November 8, 2012 - Transcript (Evidence) of Proceedings - November 8, 2012. Accessed on 19–07–2013.

Government of Canada (2012b). Study on prescription pharmaceuticals in Canada - Issue No. 26, November 21 and November 22, 2012 - Transcript (Evidence) of Proceedings - November 21, 2012. Accessed on 19–07–2013.

Government of Canada (2012c). Study on prescription pharmaceuticals in Canada - Issue No. 24, October 31 and November 1, 2012 - Transcript (Evidence) of Proceedings – October 31, 2012. Accessed on 19–07–2013.

Government of Canada (2012d). Jobs, Growth and Long-Term Prosperity: Economic Action Plan 2012, The Budget Speech [ PDF (298 KB) - external link ]. Accessed on March 28, 2012.

Government of Canada (2011). The Next Phase of Canada’s Economic Action Plan: Low Tax Plan for Jobs and Growth [ PDF (4.6 MB) - external link ]. Accessed on March 28, 2012.

Government of Canada (2007). Generic Drug Sector Study. Accessed on 11–07–13.

Government of Canada (2011). Bill C-13: The Canadian Institutes of Health Research Act. Accessed on 19–07–2013.

Hazell, L.; and, Shakir, S. A. (2006). “Under-reporting of adverse drug reactions: a systematic review”. Drug Safety, 29, 5, 385-396.

Health Council of Canada (2010). Keeping an Eye on Prescription Drugs, Keeping Canadians Safe: Active Monitoring Systems for Drug Safety and Effectiveness in Canada and Internationally [ PDF - external link ]. Accessed on 11–07–2013.

Health Canada (2006). Canada’s Food and Drugs Act & Regulations. Accessed on 07–12–2012.

House of Commons (2012). Drug Supply In Canada: A Multistakeholder Responsibility: Report of the Standing Committee on Health [ PDF (579 KB) - external link ]. Accessed August 2, 2013.

Industry Canada (2009). Mobilizing Science and Technology to Canada's Advantage: Progress Report 2009 [ PDF (528 KB) - external link ]. Accessed February 17, 2012.

Industry Canada (2007). Mobilizing Science and Technology to Canada’s Advantage [ PDF (468 KB) - external link ]. Accessed March 27, 2012.

Lexchin, J., Wiktorowicz, M., Moscou, K, & Eggertson, L. (2013). “Prescription Drugs and Administrative Costs: Provincial Drug Plan Officials’ Views of the Canadian Drug Safety System”. Journal of Health Politics, Policy and Law, 38, 3, pp. 545-571.

McDavid, J C. and Hawthorne, L.R.L. (2006). Program Evaluation and Performance Measurement: An Introduction to Practice. Thousand Oaks, CA: Sage Publications.

Morgan, S; Smolina, K; Mooney, D; Raymond, C; Bowen, M; Gorczynski, C; and, Rutherford, K (2013). The Canadian Rx Atlas, Third Edition. The Centre for Health Services and Policy Research: The University of British Columbia. Accessed on January 14, 2014.

One World Inc. (2005). Working Conference on Strengthening the Evaluation of Real World Drug Safety and Effectiveness: Summary Report. Accessed on 07–08–2012.

Research Canada (2012). Research Canada: An Alliance for Health Discovery Health Research Omnibus Survey Results.

Research Canada (2010). Canada Speaks! 2010: Canadians Go for Gold in Health and Medical Research. Accessed on 18–12–2012.

The Pulse (20 March 2013) - Accessed on 09–07–2013.

Treasury Board of Canada Secretariat (2012). Guidance on the Governance and Management of Evaluations of Horizontal Initiatives. Accessed on 07–08–2012.

Vaidya, S. S.; Guo, J. J.; Heaton, P. C.; and, Steinbuch, M. (2010). “Overview and Comparison of Postmarketing Drug Safety Surveillance in Selected Developing and Well-Developed Countries”. Drug Information Journal, 44, 519-533.

Wiktorowicz, M; Lexchin, J; and, Moscou, K. (2010). “Pharmacovigilance in Europe and North America: Divergent Approaches”. Social Science & Medicine, 75, 165-170.

Date modified:

2015-07-08