Vanier Canada Graduate Scholarships Program Evaluation
Final Report

September 2020

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

This publication was produced by the Canadian Institutes of Health Research. The views expressed herein do not necessarily reflect those of the Canadian Institutes of Health Research.

Acknowledgements

Special thanks to all participants in this evaluation – tracer survey and end of award report respondents, and key informant interview participants. Also, thank you to those who supported the evaluation: Janice Remai and Anthony da Ros (Goss Gilroy Inc.), Don Ference and Sarah Boorman (Ference & Company), Michael Beaupre (Canadian Institutes of Health Research – CIHR), the Vanier Banting Secretariat (VBS) and CIHR Funding Policy and Analytics Unit. Additional thanks to the members of the Vanier CGS Evaluation Advisory Committee – Susan Morris (Natural Sciences and Engineering Research Council of Canada – NSERC and Social Sciences and Humanities Research Council of Canada – SSHRC), Shawn McGuirk (NSERC), François Zegers and Katherine Bock (VBS), Amy Arnold, Doaa Saddek and Solange Tuyishime (Innovation, Science and Economic Development Canada – ISED).

The Vanier CGS Evaluation Team

Kwadwo (Nana) Bosompra, Ellie Fletcher, Jonathan Gilbert, Kimberly-Anne Ford, Angel Mackenzie, Shevaun Corey, Michael Goodyer, and Ian Raskin.

For more information and to obtain copies, please contact: Evaluation@cihr-irsc.gc.ca.

Table of Contents

Table of Figures

Executive Summary

Program Overview

The Vanier Canada Graduate Scholarship (CGS) program was launched by the Government of Canada in 2008 to strengthen Canada’s ability to attract and retain the world’s top doctoral students and establish Canada as a global centre of excellence in research and higher learning. The Vanier CGS award is worth $50,000 per year for over a three year period, and is available to Canadian citizens, permanent residents and foreign citizens pursuing doctoral studies at eligible Canadian institutions. There are up to 166 new awards annually, with a total of 500 active awards at a time. The program is administered by the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC) and has a secretariat housed within CIHR.

Evaluation Objective, Scope and Methodology

The objective of this evaluation is to provide Tri-agency senior management with valid, insightful and actionable findings regarding the needs addressed by the program, the effectiveness of program design in supporting outcomes, and the achievement of expected results. The evaluation covers the period from 2013-14 to 2017-18 and is the second evaluation of the program. The evaluation used multiple lines of evidence including analyses of documents, end of award reports and other administrative data, a tracer survey and, key informant interviews. The evaluation meets the requirements of the Treasury Board of Canada Secretariat (TBS) under the Policy on Results and the Financial Administration Act.

Key Findings

Relevance

There is a continued need to support top doctoral students (both domestic and international) in Canadian institutions based on Government of Canada priorities related to attracting international students to Canada and supporting the next generation of researchers. Canadian universities have seen steadily increasing international student enrollment rates, with international students representing 14% of total enrollments in 2017-18. The Vanier CGS program is intended to respond to the specific need to attract and retain world-class doctoral students by supporting students who demonstrate a high standard of scholarly achievement and leadership skills. This intended need differentiates the program from other federal government doctoral scholarship programs and is reflected in its key aspects of the program’s design; specifically, foreign citizens are eligible to apply in addition to permanent residents and Canada citizens, leadership is an explicit selection criterion, and it offers a higher award amount ($50K/year). The expected results are aligned with the Acts and research capacity building priorities of the federal granting agencies.

There was clear uptake of the Vanier CGS program by both trainees and institutions. Eligible institutions received an average of 1,084 applications annually for the period 2013-14 to 2016-17 of which, an average of 49% were nominated to the federal granting agencies. Institutions fulfilled 91% of their available nominations quota during the 2015-2018 allocation cycle.

Despite the specific nature of the program’s design, alignment with priorities and clear uptake, the ongoing challenges encountered by the program to effectively achieve its objectives means that it is not clear that the Vanier CGS program, as currently designed and delivered, is an effective means to fulfill the need to support top doctoral students (both domestic and international).

Design and Delivery

There are several aspects of the current design and delivery of the Vanier CGS program that  continue to limit the achievement of two key immediate expected results; namely, to attract to Canada top students from other countries and retain Canada’s top doctoral students at time of application. The program’s quota and nomination model limits the population of potential applicants to doctoral students affiliated with the 57 eligible institutions and can only be held at the nominating institution in Canada. Evidence indicates that a lack of harmonization between institutions’ nomination process and the Vanier CGS program’s selection process persists. Specifically, that institutions are typically hesitant to nominate candidates until they have been enrolled in a doctoral program and can be assessed by academic supervisors.

In terms of the attraction of top students to Canada, foreign candidates are underrepresented at the nomination and funding stages with only 1.5% of funded applications coming from candidates outside of Canada. In addition, despite consistent nomination rates for foreign candidates living in Canada, the success rate decreased from 38% to 23% between 2013-14 and 2017-18.

The assessment of the leadership of applicants remains a challenge due to subjectivity causing difficulty in the interpretation of and inconsistency in the assessment of the leadership criterion. This poses a particular challenge for foreign applicants, whose individual experience and understanding of leadership may vary greatly from students from different cultural backgrounds. For example, foreign applications to the Vanier CGS award were scored significantly lower on the leadership criterion than Canadian citizens and permanent residents. Similarly, foreign applications were also scored lower on academic excellence and final scores.

In terms of equity, diversity and inclusion (EDI) factors, overall no barriers were reported related to Indigenous status, being a visible minority, or a person with a disability. With respect to gender, women were significantly more likely to experience barriers related to gender than men. Given the small sample sizes associated with some of these categories, as well as the findings related to the assessment of leadership and selection of international students, further study is needed to assess EDI factors overall. This work will be important to help ensure the equitable assessment and selection of Vanier recipients and support commitments made by CIHR, NSERC and SSHRC in the Tri-Agency Statement on Equity, Diversity and Inclusion.

The Vanier CGS program is being delivered in a cost efficient manner, with a low percentage of direct administrative costs to total program expenditures (between 1.7% and 1.9%).

Performance

The Vanier CGS program is not achieving key immediate expected outcomes. The evaluation found that almost all recipients were already enrolled at eligible institutions at the time of application. The program is not effectively attracting international students to Canada as almost all international students were already living in Canada at the time of application. The program is not effectively retaining domestic students in Canada at the time of application as the majority of recipients were already enrolled in a doctoral program in Canada or would have enrolled in the same doctoral program had they not received the award.

Vanier recipients are considered to be top-tier by their academic supervisors and the majority of selection committee members interviewed. Vanier recipients are engaging in research, teaching and service leadership opportunities. They are also establishing national and international collaborations during the award, many of which continue after the award, but there were minimal differences between recipients and applicants with regard to collaborations following their doctoral degree.

Although the Vanier CGS program is achieving its expected intermediate outcomes, the evaluation found that applicants (who did not receive the award) report very similar outcomes, which calls into question the incremental value of the program. In terms of retention post-graduation, the majority of both Vanier recipients and applicants are living and working in Canada, working in the academic sector, and in research intensive careers with no observable differences. Among Vanier recipients, Canadian citizens were more likely to be working in Canada (80%) than permanent residents (66%) and foreign citizens (49%). Recipients have higher research productivity (significantly more peer reviewed publications and international conference presentations) than applicants following completion of their doctoral degrees. Vanier recipients are more likely to be in more advanced academic positions (e.g., assistant or associate professor) compared to applicants (e.g., postdoctoral fellow), but there are no differences in tenure status and time to tenure. There are no differences between recipients and applicants in terms of funding received (e.g., postdoctoral awards, research grants) since completing their doctoral studies. There were few differences between the two groups in terms of the availability and impact of leadership development opportunities during their doctoral degrees.

Given that key design and delivery aspects continue to limit the achievement of intended outcomes, especially the attraction and retention of top doctoral students at time of application, there is a need to change the design and delivery of the program to better achieve the current objectives and/or revise the objectives to better align with the current delivery model. Further, the evidence indicating the limited incremental impact of the program on recipients when compared to applicants in the areas of leadership opportunities, post-graduation retention and career outcomes reinforces the need to reconsider the program’s design, delivery and objectives in relation to the Canada Graduate Scholarships - Doctoral program as well as agency-specific doctoral scholarship programs.

Recommendations

The evaluation makes the following recommendations aimed at improving the ongoing implementation and performance of the program.

Recommendation 1:

The Vanier CGS program needs to change its current objectives and/or design and delivery model in order to more effectively attract top doctoral students to Canada and retain top domestic students in Canada at time application. Given the similar levels of achievement by recipients and applicants related to leadership opportunities, post-graduation retention and career outcomes, the changes to the Vanier CGS program need to be made with due consideration of other federal doctoral scholarship programs.

Recommendation 2:

In light of the evaluation findings and the Tri-Agency Statement on Equity, Diversity and Inclusion, the Vanier CGS program should examine the nature and extent of EDI barriers, including GBA+ analysis, related to the potential biases associated with the leadership selection criteria, the nomination process, and the selection of international doctoral students in order to more equitably assess and select Vanier recipients.

Program Profile

Program Description

The Vanier CGS program was launched by the Government of Canada (GOC) in 2008 to strengthen Canada’s ability to attract and retain the world’s top doctoral students and establish Canada as a global centre of excellence in research and higher learning.Footnote 1 The expected results of the program as outlined in the program’s Terms and ConditionsFootnote 2 are:

  1. Establishment of a Canadian doctoral scholarship that is internationally competitive and internationally recognized;
  2. Enhanced capacity of Canadian universities to attract the best and brightest students from Canada and the world with the potential to become leaders in the next generation of researchers in Canada;
  3. Attract to Canada top students from other countries;
  4. Retain Canada’s top doctoral students in the face of intense international competition for the brightest researchers;
  5. Promote and brand Canadian universities as world-class institutions of research and higher learning;
  6. Establish effective research collaborations and networks; and,
  7. Create alumni of “ambassadors” to promote Canada as a destination for research excellence and higher learning.

The program’s updated logic model was approved in 2016 (Figure 1: Vanier CGS Logic Model). The Vanier CGS is intended to complement other federal level doctoral support (e.g., Canada Graduate Scholarship - Doctoral program) and is delivered through the three federal granting agencies, referred to as the Tri-agencies – Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC), and the Social Sciences and Humanities Research Council (SSHRC).Footnote 3

The Vanier CGS award is worth $50,000 per year for three years, and is available to Canadian citizens, permanent residents, and foreign citizens pursuing doctoral studies at eligible Canadian institutions.Footnote 4,Footnote 5 Canadian citizens and permanent resident holders of Vanier scholarships may also apply for a Michael Smith Foreign Study Supplement of up to $6,000 to help offset the costs of undertaking research studies outside Canada for a period of three to six months during their degree.

Up to 166 Vanier scholarships are awarded each year and are divided among the Tri-agencies who each offer 55 awards annually, with the additional one award rotated annually between NSERC and CIHR. Apart from the first three years of the program when expenditures were ramping up, investments in the program have remained steady at approximately $25 million annually since 2011-2012 with a total of up to 500 scholarships active at any time. As of 2017-18, a total of 1,659 scholarships have been awarded through the Vanier CGS program and over $200 million has been invested in the program since its inception in 2008-09. The Vanier-Banting Secretariat (VBS), which is housed within CIHR, is responsible for the day-to-day administration of the program.

Application and Review Processes

Vanier scholarships are awarded after a competitive selection process involving reviews conducted at institutions as well as by the Tri-agencies.Footnote 6 Students must be nominated for a Vanier CGS award by an eligible Canadian institution that holds Vanier CGS quotas allocated by one or more of the Tri-agencies. The scholarship liaison officer at each eligible Canadian institution is responsible for coordinating their internal institution review to determine the selected candidates and forwarding those nominations to the Vanier CGS program in accordance with their institution’s quota. Internal review practices vary across institutions, for example, in some institutions the scholarship liaison officers send mass emails to all graduate students highlighting the timelines for the Vanier scholarship, while in others, potential candidates must be invited to apply by their supervisor or head of department. Three agency-specific selection committees then evaluate and recommend the top 55 or 56 candidates from their respective domains (for a total of up to 166 candidates across the three agencies) to the Tri-agency Programs Steering Committee (TAP-SC). The TAP-SC, which comprises the Presidents of the Tri-agencies and the Deputy Ministers of Innovation, Science and Economic Development Canada (ISED) and Health Canada, makes the final funding decisions. Further details of the review process are available on the program’s website.

Description of Evaluation

Evaluation Purpose and Scope

This evaluation covers the period from 2013-14 to 2017-18 and is the second evaluation since the program’s launch in 2008. The purpose of this evaluation is to provide Tri-agency senior management with valid, insightful and actionable findings regarding the:

The first evaluation, completed in 2014, found that the program was not meeting its key outcomes of attracting top-tier doctoral students from outside Canada or retaining top-tier domestic students (Canadian citizens and permanent residents). Therefore for this evaluation, a decision was made to build on the first evaluation and focus primarily on the key outputs and outcomes expected to occur within ten years of program inception. Assessment of the expected outcome of increased national and international awareness of the Vanier CGS was also scoped downwards since it would have required a larger international study. The evaluation meets the Tri-agencies’ requirements to the Treasury Board of Canada Secretariat (TBS) under the Policy on Results and related instruments and also aligns with section 42.1 of the Financial Administration Act. The evaluation is a regularly scheduled Tri-agency evaluation included in the approved 2018-19 evaluation plans of CIHR, NSERC and SSHRC and is led by the CIHR’s Evaluation Unit within the Office of Audit and Evaluation in collaboration with the NSERC-SSHRC Evaluation Division.

Previous Evaluation

In 2014, the CIHR Evaluation Unit conducted an evaluation of the first five years of the Vanier CGS program which included a review of competitions funded from 2008-09 to the end of the fiscal year 2012-13. The evaluation examined the program’s relevance, design and delivery, and performance.

Evaluation findings indicated that the program was supporting world-class doctoral students – the majority of whom were Canadian, with most Vanier scholars demonstrating exceptional leadership in the area of research. The majority of Vanier scholars were satisfied with their training and skill development and most reported that their training had been useful in preparing them for their career. The majority of the Vanier scholars who had completed their studies were employed in the university sector and were living in Canada.

The evaluation found that the program’s design and delivery was inhibiting its ability to attract students from outside of Canada and retain Canadian students and was also limiting the population of candidates put forward for nomination. The interpretation and assessment of the leadership criterion was identified as a challenge. There were also questions about the incremental outcomes associated with the higher value of the scholarship in comparison to the Canada Graduate Scholarships - Doctoral (CGS-D) program and other Tri-agency doctoral awards.

The evaluation made four recommendations: 1) change the allocation and application processes to better meet the program’s objective of attracting and recruiting world class doctoral students to Canadian universities; 2) establish a clearer definition of what leadership is and how it should be assessed, especially in regard to foreign students; 3) ensure that Vanier CGS graduates are connected to the program and to other scholars after completion of their studies; and, 4) undertake a further analysis to assess the similarities and differences in the results achieved across doctoral award programs and consider the results of this analysis in the future planning and design of the program.

As of November 2019, recommendation 1 remained partially implemented with work completed to move from a three-year to an annual allocation process and to consider changes to the application timelines, but work is ongoing to examine the barriers to and means by which to increase international participation. For recommendation 2, actions were taken to continue to clarify the definition of leadership including instructions to applicants and reviewers. In the case of recommendation 3, social media pages and distribution lists were created to support the Vanier Alumni Network. And finally for 4, a study directly comparing items common to both the Vanier CGS and CGS evaluations was completed in 2016 and revealed that each program can produce certain advantages however the causes of the differences are unclear. The Vanier CGS seems to have advantages in relation to financial support, debt reduction, research papers and skills related to the larger community; whereas, the doctoral component of the CGS is perceived to be advantageous in relation to pace of progress, rate of completion, presentations at international conferences and usefulness of training.

Evaluation Methodology

Evaluation Approach

The evaluation employed both quantitative and qualitative approaches to data collection and analyses. Consistent with TBS guidance and recognized best practice in evaluation,Footnote 7 multiple lines of evidence were used to triangulate evaluation findings: document review, analysis of administrative data on program applicants, analysis of Vanier End of Award Reports (VEAR) completed by award recipients, and a survey of the first four cohorts of scholarship recipients and applicants (referred to in this report as a “tracer survey”). There were also key informant interviews with scholarship recipients (n = 14) and applicants (n = 12), scholarship liaison officers (n = 4), academic supervisors (n = 5), Vanier CGS selection committee chairs and members (n = 7), executive and senior program management (n = 5), and a representative from Global Affairs Canada (GAC) (n = 1). Please note that for the purposes of this report, recipients are defined as those who received a Vanier CGS; whereas, applicants are defined as those who applied for and/or were nominated but did not receive, or declined, a Vanier CGS. As outlined below, Vanier CGS applicants may have, and more than often not did, receive other funding scholarship funding for their doctoral degree.

Evaluation Questions

The evaluation addresses the following specific questions.

Relevance
  1. Is there a continued need for the Vanier CGS program and is the program aligned with federal government priorities?
    • 1.1. Is there an identified need for support for top doctoral students in Canadian institutions?
    • 1.2. Is the Vanier CGS program aligned with GOC and federal granting agencies’ roles and priorities?
Design and Delivery
  1. Does the design and delivery of the Vanier CGS program support achievement of intended outcomes?
  2. Has the Vanier CGS program been delivered by the federal granting agencies in a cost efficient manner?
Performance
  1. Is the Vanier CGS program achieving its expected outputs and immediate outcomes?
    • 4.1. Is the program attracting top international students?
    • 4.2. Is the program retaining top Canadian students?
    • 4.3. Are recipients demonstrating research, academic and service leadership?
    • 4.4. Are recipients establishing national and international collaborations?
  2. Is the Vanier CGS program achieving its expected intermediate outcomes?
    • 5.1. Have Vanier CGS alumni remained in Canada?
    • 5.2. Have Vanier CGS alumni undertaken research careers in Canada?
    • 5.3. Have Vanier CGS alumni become leaders in early career research?

In addition to analyses addressing the core evaluation issues, the evaluation also explored issues related to Equity, Diversity and Inclusion (EDI) and Gender-Based Analysis Plus (GBA+) by examining the survey sample in terms of such characteristics as sex, gender, Indigenous status, being a visible minority or a person with a disability, as well as by asking recipients and applicants if they had experienced barriers related to these factors during their Vanier CGS. Overall, no differences were observed except, in some instances, for sex and those are noted within the report. Evaluation findings were disaggregated by relevant population variables including funding status (recipient and applicant), agency (CIHR, NSERC, and SSHRC), sex,Footnote 8 and citizenship status (citizen or permanent resident of Canada vs. citizen or permanent resident of a foreign country). Significant differences are noted within the report. Additional details about the methodology are provided in the Appendix (Appendix B - Methodology).

Limitations of this Evaluation

Most evaluations face limitations that have implications for the validity and reliability of the findings. The following outlines the key limitations encountered and the mitigation strategies used to help ensure the evaluation results can be used with confidence to inform program decision making.

It was not possible to attribute recipients’ achievements solely to the Vanier scholarship given that over their career trajectory they received additional support at the doctoral level and/or support at other levels from different funders within Canada and abroad. The evaluation therefore interpreted any findings on recipients’ outputs and outcomes in terms of the contribution that the Vanier CGS program has made.

The evaluation did not directly compare the Vanier CGS in relation to other federal level doctoral support programs but instead comparisons were made between scholarship recipients and applicants who did not receive the scholarship for two reasons. First, in response to Budget 2018 commitments, ISED had initiated a study of the suite of federal level scholarships including doctoral awards and the TBS was leading a review of skills programming.Footnote 9 Second, students who apply for the Vanier CGS would also typically apply to other Tri-agency programs (based on their eligibility). Therefore, the majority of Vanier applicants who do not receive the Vanier CGS end up receiving either the CGS-D or agency-specific doctoral awards with several also receiving awards from provincial government organizations and Canadian universities. Indeed, 95% (n = 247) of applicants in the survey sample reported receiving funding for their doctoral studies. Of these, 67% (n = 236) indicated at least one of the Tri-agencies as their source of funding, 36% mentioned provincial government organizations, and 54% mentioned Canadian universities (respondents could mention more than one source). Of the 170 applicants who identified a specific Tri-agency program, almost half received a CGS-D scholarship (44%) and less than a quarter received the SSHRC Doctoral Fellowship (19%), the CIHR Doctoral Research Award (12%), the NSERC Postgraduate Scholarship-Doctoral (11%), or other Tri-agency awards. Thus, the comparison with applicants is essentially a comparison with recipients of other federal level doctoral award programs. It should be noted that the results from the Vanier End of Award Reports, following completion of the award are based on self-report data from recipients and there was no comparison group for these outcomes.

The response rate for Vanier applicants in the tracer survey was low (25%); and a comparison of population and sample characteristics on the basis of applicant status, agency, gender, citizenship status and year of application showed that there was an underrepresentation of applicants in the survey sample (49%) as compared to the study population (66%) and correspondingly an overrepresentation of recipients in the sample (51%) as compared to the population (34%). There was also an overrepresentation of Canadian citizen applicants in the survey sample (73%) as compared to the study population (66%) and an underrepresentation of foreign citizen applicants (19%) as compared to the study population (27%). Therefore, the data were weighted for application status and citizenship status (applicants only); however, it should be noted that in spite of weighting to these known population characteristics, non-response bias on the basis of other characteristics of the population may be present.

For key informant interviews, apart from Vanier recipients (n = 14) and applicants (n = 12) the numbers of interviewees in the other groups were small (scholarship liaison officers, n = 4; academic supervisors, n = 5; selection committee members, n = 4; selection committee chairs, n = 3; senior program management, n = 1 and a GAC representative, n = 1) and it is possible that the discussions may not have reached saturation in the findings within groups. To mitigate this, some of the very small groups were merged based on the nature and affiliation of respondents, for example selection committee members and chairs were treated as one group (given that they were providing perspectives from peer review committees) and steering committee members, senior program management and the GAC representative were also combined (given that they were providing perspectives on the overall design and delivery of the program, although senior program management also provided information on program performance). Additionally, the interview questions were very similar especially for the merged groups, thus a congruence of findings across groups could be argued to be reflective of saturation. Furthermore, the use of multiple lines of evidence including the tracer survey and end of award reports allowed for a triangulation of findings, further mitigating this risk.

Evaluation Findings

Relevance

Key Findings

GOC has identified a continued need to support top doctoral trainees

Through its policies and publications, the GOC has demonstrated a continued need to support the next generation of researchers as well as to attract international students to Canada and increase international placements for domestic students. Since its launch in 2008, the Vanier CGS program aims to strengthen Canada's ability to attract and retain world-class doctoral students and establish Canada as a global centre of excellence in research and higher learning, attracting an annual budget of $25 million.

International student enrollment rates have been steadily increasing in Canadian universities over the past two decades, with international students representing 14% of total enrollments in 2017-18 (Statistics Canada, 2018). Specifically the total number of international students has risen from 199,077 in 2013-14 to 296,469 in 2017-18, with increases of 11% in 2016-17 and 13.5% in 2017-18. According to Statistics Canada (2018) increases in international student enrolments in Canada are due to a variety of reasons, including programs and policies put in place to increase their numbers, the quality of postsecondary education in Canada, and the appeal of Canada as a study destination.

The Vanier CGS aims to fulfill a particular need within the GOC’s doctoral training support landscape

The Vanier CGS aims to fulfill a particular need within the GOC’s doctoral training support landscape as compared to other federal level doctoral awards available through the Tri-agencies, which include the CGS-D, the CIHR Doctoral Foreign Study Award (DFSA), the SSHRC Doctoral Fellowship, and the NSERC Postgraduate Scholarships-Doctoral (PGS-D). The key differences between the Vanier and the other awards are: the award amount, citizenship eligibility criteria, requirement to take up the award at the nominating Canadian institution, explicit inclusion of leadership as an assessment criterion, and the number of scholarships awarded annually (see Figure 2: Comparison of doctoral scholarship programs offered by federal granting agencies ). The Vanier CGS award is worth the highest amount at $50K per annum, the rest range from $20K-$35K. The Vanier CGS is the only graduate-level award open to foreign citizens; however, all agency-specific awards allow the option for trainees to study abroad. The Vanier CGS is the only one that specifically includes leadership as a selection criterion. The majority of federal level doctoral support is provided via the CGS-D with a total of approximately 2,500 students holding the award annually, at the time of this analysis Investments made in Budget 2019 will increase the total number of awards to 3,000.

The Vanier CGS was highlighted in Seizing Canada’s Moment: Moving Forward in Science, Technology and Innovation as being among “the world’s most prestigious awards for doctoral students” and as one of the vehicles for “strengthening the ability of Canadian universities to attract and retain world-leading research talent and support the development of the research leaders of tomorrow.”Footnote 10 Budget 2019 reiterated this position by highlighting one of the goals of the International Education Strategy – “Attract more top-tier foreign students to Canada by promoting Canadian educational institutions as high-calibre places to study.” The government’s commitment to supporting talent development was further evidenced in the establishment of the Canada Research Coordinating Committee (CRCC) whose first published work plan included “establishing Canada as a world leader in supporting the development of talent throughout the research career life cycle.”Footnote 11 More recently, in its first progress report, Strengthening Canadian Research: Progress Report 2018-19, the CRCC re-emphasized the importance of developing the researcher pipeline by highlighting several steps taken to support early career researchers.Footnote 12

Furthermore, Canada’s Fundamental Science Review examined the Vanier CGS and other scholarships and fellowships available in the graduate training space in the research career cycle and recommended recruiting more international students, and increasing international placement opportunities for domestic students.Footnote 13

When this evaluation commenced in 2018, ISED had already initiated a study, in response to the Budget 2018 call for further work to determine how to better support students as the next generation of researchers. The study planned to look at gaps and opportunities within the suite of federal level scholarships and cover themes such as award value, duration, portability, and EDI. At the same time, the TBS was leading a Horizontal Review of Skills Programming (HRSP) that related to a Budget 2018 commitment to undertake such a review over the following year to ensure that Canadian workers had the skills required to meet changing labour market needs – skills needed to succeed in an evolving economy.

There is clear uptake of Vanier CGS awards by trainees and institutions

Analysis of program administrative data on the most recently completed quota allocation cycle (2015-2018) show that the eligible institutions comprised 57 Canadian institutions that received at least one nomination allocation from the Tri-agencies for the 2015-2018 allocation cycle and together they fulfilled 91% of the 1800 available nominations. About one-third of the institutions (30%) fulfilled their quotas while 12% did not submit any nomination. Additional analysis of numbers of candidates passing through each of the stages from application at the institutional level, through nomination by institutions to the Tri-agencies and recommendation for a scholarship by the Tri-agency selection committees confirmed a robust demand for the scholarship across the institutions. Between 2013-14 and 2016-17, the eligible institutions together received between 975 and 1,148 applications (M = 1,084) annually and nominated, on average, 49% of applicants. At the end of the process, an average of 15% of applications (or 31% of nominations) were awarded the scholarship (Figure 3: Vanier CGS application outcomes by stage of application).

When looking at the proportion of applications by citizenship status across agencies, lower application rates for foreign citizens was observed for all agencies. However, NSERC and SSHRC had a greater proportion of foreign applications than CIHR.

The proportion of nominations recommended for funding (relative to application rates) was fairly equal for both Canadian and foreign applicants across the tri-agencies; however, the proportions varied across years more so for CIHR and SSHRC (i.e., some years foreign applicants had slightly higher rates, while some years Canadian students had higher rates) than NSERC. Foreign applicants to NSERC had lower success rates more consistently across years. In terms of the proportion of successful Canadian versus successful foreign applicants, there was greater disparity for CIHR (than NSERC and SSHRC).

Figure 3: Vanier CGS application outcomes by stage of application

Figure 3 – Long description

Vanier CGS application outcomes by stage of application

Applied but not nominated Nominated but not funded Funded Total
2013-14 507 302 166 975
2014-15 556 407 166 1129
2015-16 550 366 166 1082
2016-17 598 384 166 1148
2013-17 2211 1459 664 4334

Vanier CGS application outcomes by stage of application

Applied but not nominated Nominated but not funded Funded
2013-14 52% 31% 17%
2014-15 49% 36% 15%
2015-16 51% 34% 15%
2016-17 52% 33% 14%
Overall 51% 34% 15%

Source: Vanier Banting Secretariat administrative data, 2018.

Additionally, the majority of scholarship recipients (65%, n = 348) irrespective of area of study or sex had already received other offers of doctoral support at the time they were notified of the Vanier CGS award. Trainees from foreign countries (55%, n = 103) were less likely than Canadians and permanent residents (69%, n = 245) to have received another offer at that time. Furthermore, very few successful candidates declined the award between 2013-14 and 2017-18 (2% out of 831), with the most commonly reported reason being having accepted another award (n = 3).

Program stakeholders support funding top-tier doctoral trainees

Interviews with senior Tri-agency officials, supervisors of Vanier scholarship recipients, Vanier CGS selection committee members, scholarship liaison officers from the universities, and former scholarship recipients and applicants identified a need to support top doctoral students in Canadian institutions. They uniformly agreed that Canada needs to attract and retain world-class foreign and Canadian doctoral trainees in order to support a strong Canadian research eco-system that has the skills and innovation to remain competitive on a global scale.

All senior Tri-agency officials (5/5) felt that delivering the Vanier CGS program (as a means to support doctoral trainees) is an appropriate role for the federal government. In their view, the federal government is mandated to support the entire research ecosystem in Canada from undergraduate trainees to established researchers and therefore well positioned to invest in doctoral trainees. The majority of interview respondents from the universities (15/17) were of a similar view that the program aligns closely with the federal government’s priorities to support high-quality scientific research in Canada and enhance the future economic, social and political prospects of the country. They believed that federal funding ensures the program is recognized on a larger scale, both Canada-wide and globally.

Almost all senior Tri-agency officials (4/5) saw the Vanier CGS program as complementing other trainee funding programs and considered it unique due to the high monetary value of the award, the eligibility of international students to apply, the inclusion of leadership as a criterion and its prestige due to the low number of awards available. At the same time, most officials (3/5) acknowledged an opportunity to examine these key differences in terms of their benefits and challenges, assess any gaps in funding, and harmonize across the awards to create a suite of programs that function well together (e.g. allow all awards to be portable, provide the same amount of funding for each award, create one program for doctorate awards).

The majority of academic supervisors (4/5) and approximately one quarter of selection committee members (2/7) expressed the view that the program overlaps or takes funding away from the CGS-D program and that the same objectives could be achieved by reducing the amount of the Vanier scholarship and increasing the number of Vanier scholarships awarded.

Vanier CGS objectives are aligned with Tri-agency priorities

The Vanier CGS program aligns with the corporate mandates and priorities of the Tri-agencies and the program’s objectives match with their functions to promote and assist research in their respective domains outlined in their respective Acts: Canadian Institutes of Health Research Act, 2000; the Natural Sciences and Engineering Research Council Act 1985; and the Social Sciences and Humanities Research Council Act, 1985.

The CIHR Act (S.C. 2000, c6) acknowledges the importance of the “attraction of the best health researchers in Canada and the world and their development and retention in Canada” and its aims include, among other objectives, “building the capacity of the Canadian health research community through the development of researchers and the provision of sustained support for scientific careers in health research.”Footnote 14 CIHR has consistently supported programs to train the next generation of researchers in all its strategic plans. For the strategic plan in place during the period of the evaluation, Roadmap II, under “Strategic Direction 1: Promoting Excellence, Creativity and Breadth in Health Research and Knowledge Translation” and Section “1.2: Building a Solid Foundation for the Future,” the Vanier CGS program is one of two programs highlighted by CIHR as key vehicles for training and mentoring the next generation of researchers and professionals and positioning them for success in the health-related academic and professional careers of the future.Footnote 15

Secondly, health research training is a core priority for CIHR as stated in the agency’s action plan for training, a strategy that aims “to generate scientific, professional, and organizational leaders within and beyond the Health Research Enterprise.”Footnote 16 Thirdly, CIHR’s current, 2019-20 Departmental Plan (formerly Reports on Plans and Priorities), specifically mentions Vanier CGS awards as one of the means to help achieve “Departmental Result 2 – Canada’s health research capacity is strengthened.”

NSERC’s enabling legislation, the Natural Sciences and Engineering Research Council Act, identifies as one of two core functions of the agency, “to promote and assist research in the natural sciences and engineering, other than the health sciences.” To support this function, the NSERC’s Departmental Plans have underlined the importance of developing the next generation of scientists and engineers. The 2019-20 Departmental Plan reaffirms the agency’s core responsibility as through grants, fellowships and scholarships, “promotes and supports research and research training in the natural sciences and engineering to develop talent, generate discoveries, and support innovation in pursuit of economic, environmental and social outcomes for Canadians.”

Under the Departmental result “Canada has a pool of highly skilled people in the natural sciences and engineering,” NSERC invests in grant, scholarship and award funding to support “the attraction, retention and development of highly qualified and skilled people in National Science and Engineering in Canada.”Footnote 17

Similar to the two other federal granting agencies, the SSHRC Act establishes that a key function of the agency is to “promote and assist research and scholarship in the social sciences and humanities.” The current Departmental Plan (2019-20) reiterates that the core responsibility of SSHRC is to promote and support “research and research training in the social sciences and humanities to develop talent, generate insights and build connections in pursuit of social, cultural and economic outcomes for Canadians”,  through grants, fellowships and scholarships. The 2019-20 Departmental Plan stipulates that to achieve its Departmental Result 2, “Canada has a pool of highly skilled people in the social sciences and humanities,” the SSHRC provides funding for scholarships, fellowships and research grants in the social sciences and humanities.

Additionally, the SSHRC’s current strategic plan (2016-2020), reaffirms the importance of research training in its vision statement: “Canada sustains and enhances its position as a global leader in humanities and social sciences research and research training, improving the lives of Canadians through ideas and innovation.”Footnote 18 The research training objective is promoted through the Talent umbrella program the goal of which is “to support students and postdoctoral researchers in order to develop the next generation of researchers and leaders across society, both within academia and across the public, private and not-for-profit sectors.”

Senior Tri-agency officials interviewed acknowledged the Vanier CGS program’s alignment with the Federal government’s priorities (4/5) noting that the Tri-agencies, along with the CRCC, were jointly interested in promoting collaboration and harmonization around improving the research ecosystem and in supporting the training and career development of researchers.

Design and Delivery

Key Findings

Recipients are satisfied with key program elements

Scholarship recipients were generally satisfied with the main elements of the Vanier CGS program, consistent with previous evaluation findings (2014). According to end of award reports, recipients (n = 348) were satisfied with the application process (M = 3.99 out of 5, SD = 0.75), eligibility requirements (M = 4.11, SD = 0.70), decision/peer review process, (M = 4.12, SD = 0.71), length of the award (M = 4.21, SD = 0.94), and were highly satisfied with the monetary value of the award (M = 4.66, SD = 0.58).Footnote 19

All scholarship recipients interviewed (14/14) also expressed their satisfaction with the monetary value of the award and over half (8/14) were satisfied with the duration of the award, noting that the high value of the award provided them with time and resources to focus on writing and publishing, attending conferences, and engaging in activities which enhanced their leadership skills.

However, other respondents interviewed expressed contrary views about the award amount. Approximately one-quarter of selection committee members (2/7) felt that the award amount should be reduced to fund more students or redirected to provide mentoring and training opportunities (e.g., leadership or professional skill development) and slightly more than half of supervisors (3/5) felt that the high award amount had created wage disparities among graduate students, tiered research environments in laboratories, and tension among some graduate students.

Concerns were also raised about the Vanier CGS application deadlines with three-quarters of scholarship liaison officers (3/4) and approximately two-thirds of supervisors (3/5) suggesting that the timing of application deadlines should be changed to better align with university admission processes, a finding that is consistent with findings in the 2014 evaluation. In fact, as part of the management action plan for the previous evaluation, the VBS consulted institutions, reviewers, and Tri-agency representatives about their preferred timelines but consensus on date changes was not achieved and the timelines have largely remained the same.

Very few perceived barriers due to EDI factors

Recipients and applicants who participated in the key informant interviews and tracer survey respondents were asked if they had experienced barriers related to gender, Indigenous status, being a visible minority, or a person with a disability when participating in the Vanier CGS. Overall, the majority of recipients (13/14) and applicants (10/12) interviewed did not report experiencing any barriers due to EDI factors when participating in the program.

Overall, recipients and applicants surveyed indicated there were no barriers related to gender (M = 1.10 out of 5, SD = 0.43, n = 500); however, women were significantly more likely to experience barriers related to gender compared to men although the absolute difference was very small (M = 1.13, SD = 0.50 vs. 1.06, SD = 0.31). Similarly, survey respondents did not report any barriers related to Indigenous status (M = 1.00, SD = 0.00, n = 3), being a visible minority (M = 1.29, SD = 0.67, n = 90), or a person with a disability (M = 1.32, SD = 0.66, n = 38). Note that the sample sizes for some of these categories were very low (See Appendix B - Methodology for more details).

According to senior Tri-agency officials there has been increasing assessment and monitoring of the Vanier CGS program around EDI issues and there have indeed been some initial, albeit ad hoc, changes to the design and delivery of the program. The program’s website has a page that provides resources on EDI related issues to guide applicants, nominating institutions, referees and selection committee members and an online training module on bias in peer review is also available on the website for selection committee members.

Program design and delivery limits the attraction of top doctoral students at time of application

One of the expected results of the Vanier CGS program outlined in the program’s Terms and Conditions (2008) is to enhance the capacity of Canadian universities to attract the best and brightest students from Canada and the world. However, some aspects of the program’s design were perceived to be inhibiting its ability to attract top-tier doctoral students. First, the program stipulates that students can only be nominated for a Vanier CGS award by a recognized Canadian institution that holds a Vanier CGS quota allocated by one or more of the Tri-agenciesFootnote 20; there were 57 such eligible institutions out of 92 in the most recently completed allocation cycle (2015-16 to 2017-18).Footnote 21 The methodology for allocating quotas to institutions is based on the amount of funding received through the Canada Research Chairs (CRC) program and agency doctoral programs by the institution and having an eligible doctoral program. Some senior Tri-agency officials (3/5) noted in the interviews that the program’s model of identifying scholarship candidates only through university nominations/allocations restricts the applicant pool and does not guarantee that the program is selecting the best and brightest students worldwide but rather that selection is occurring among only known individuals at eligible institutions.

Second, the program requires that the scholarship can only be held at the nominating Canadian institution.Footnote 22 Although this is a deliberate design feature of the program, assumed to aid in the attraction and retention of doctoral students to Canada, it is likely that this feature further restricts the Vanier CGS applicant pool by eliminating from consideration any potential top-tier candidates who would like to pursue doctoral studies outside Canada. Additionally, the requirement does not align with the conclusions and recommendations of Canada’s Fundamental Science Review (relating in particular to Recommendation #7)Footnote 23 and Canada’s Science Vision (relating in particular to the expected outcome of making Canadian science more collaborative).Footnote 24 Recommendation #7.1 of the Fundamental Science Review which talks about harmonizing scholarship and PDF fellowship programs, is premised on the need to increase the recruitment of top-flight international graduate students and PDFs as well as the international placement of domestic students and trainees to give them opportunities to learn from international exposure to leading scientists and scholars. Requiring the scholarship to be held only in Canada does not easily align with increasing international placements for domestic trainees. Similarly, one of the expected outcomes of Canada’s Vision of Science is “making Canadian science more collaborative” and one of the pathways to achieve this is supporting research that is international, interdisciplinary, fast-breaking and higher-risk. Researchers can develop international networks through international exposure to leading scientists and scholars right from the early stages of their careers and requiring the scholarship to be tenable only in Canadian institutions may not necessarily be the best approach to achieving such an outcome.

The first evaluation of the Vanier CGS program concluded that the program had not met its objective of attracting and recruiting world-class students from outside Canada because the majority of supported students were already studying in Canada when they applied.Footnote 25 In response to those findings the VBS explored options to increase international participation, including an examination of barriers to international participation and options to increase nominations of foreign students to 50%. However, given the current delivery model increasing nominations would likely not address the issue of attracting those from outside of Canada but instead increase funding for those foreign applicants already enrolled at a Canadian institution or living in Canada.

End of award reports show that during the period under review, 61% of recipients were Canadian citizens or permanent residents when they applied for the scholarship (less than 1% were Canadians living abroad), 37% were foreign candidates living in Canada, and 1.5% were foreign candidates applying from outside of Canada. This finding was corroborated by findings from the survey which showed that almost all respondents (98% out of 508 recipients and applicants) were living in Canada at the time of nomination for the award.

End of award reports also show that the majority of recipients (80% of 348) would have enrolled in the same doctoral program had they not received the award or were already enrolled in a doctoral program (8%). Only 1% would have enrolled in another doctoral program in Canada and 4% would have enrolled in a doctoral program outside of Canada. Only 4% would not have enrolled in a doctoral program at all and only 2% would have dropped out of their current program.

Although interviewees see the eligibility of international students to apply for the Vanier CGS as a major strength of the program, some selection committee members (3/7) believe that the program is not achieving its intended outcome of attracting world-class foreign doctoral students to Canada due to the “institutional culture” surrounding the nomination process. For example, one selection committee member stated: “There’s a mismatch between what happens at the university level to identify students versus what happens at the Vanier selection level. They are not harmonized to some extent.” Foreign candidates are seen as having limited networks at Canadian institutions and institutions are typically hesitant to nominate “unknown” candidates unless they have been enrolled for at least a year to enable them have opportunities to engage with and be assessed by academic supervisors. Over half of the academic supervisors interviewed (3/5) confirmed this, stating that they were more likely to nominate students who were already studying at their institution and with whom they had a shared working experience. Here, it is important to note that this has bearing not only on foreign and international students, but also Canadian students. In addition, few scholarship liaison officers (2/5) report seeking out international candidates for the award citing lack of time, resources, accessibility, and perceived challenges relating to leadership (which is discussed in the next section of the report).

Administrative data for the program (2013-14 to 2016-17) confirms that foreign candidates were underrepresented at the nomination and recommended for funding stages when compared with the application stage. In 2013-14, for example, 40% of applications were from foreign candidates but only 31% of nominations and 33% of applications recommended for funding were from foreign candidates (Figure 4: Vanier CGS applicant type by stage of application). The data also corroborated the earlier findings related to where candidates were living at the time they applied to the program.

Figure 4: Vanier CGS applicant type by stage of application

Figure 4 – Long description
Canadian Candidate Foreign Candidate (applying from within Canada) Foreign Candidate (applying from abroad)
2016-17 Funded 0.686747 0.313253 0
Nominations 0.669091 0.325455 0.005455
Applications 0.614983 0.367596 0.017422
2015-16 Funded 0.698795 0.301205 0
Nominations 0.699248 0.293233 0.007519
Applications 0.644177 0.349353 0.00647
2014-15 Funded 0.728916 0.271084 0
Nominations 0.715532 0.277487 0.006981
Applications 0.605846 0.372896 0.021258
2013-14 Funded 0.674699 0.325301 0
Nominations 0.685897 0.311966 0.002137
Applications 0.591795 0.396923 0.011282

Source: Vanier Banting Secretariat administrative data, 2018.

Note: Canadian Candidates include those applying from abroad (<1%)

Other administrative data from the VBS also show that between 2013-14 and 2017-18, despite relatively consistent nomination rates for foreign candidates living in Canada (proportionate to application rates) success rates for foreign candidates decreased from 38% (2013-14) to 23% (2017-18) (Figure 5: Number of Tri-agency applications and success rates by citizenship status).

Consistent with findings of the 2014 evaluation, this evaluation found that the Vanier CGS program’s design and delivery continues to inhibit the program’s ability to attract and retain top-tier doctoral students.

Figure 5: Number of Tri-agency applications and success rates by citizenship status

Figure 5 – Long description
Agency Citizenship Competition Year Grand Total Within Citizenship Success Rate Recommended for funding Not recommended
Overall Canadians and permanent residents 2013-14 326 34% 112 214
2014-15 400 30% 121 279
2015-16 367 32% 116 251
2016-17 366 31% 114 252
2017-18 401 33% 133 268
Foreign citizens 2013-14 141 38% 54 87
2014-15 162 28% 45 117
2015-16 158 32% 50 108
2016-17 178 29% 52 126
2017-18 144 23% 33 111
Overall Total     2643 31% 830 1813

Source: Vanier Banting Secretariat administrative data, 2018. N = 2643.

Assessment of leadership remains a challenge

The 2014 evaluation found that the assessment of leadership was a challenge in that there was a need for a clearer definition and scoring process and in response, the VBS communicated revised guidelines for reviewers and candidates in June 2015. Currently, the applicant and selection committee member pages of the program website has cross references to each other’s pages with links to writing the personal leadership statement (in the case of applicants) and assessing leadership and the other criteria (for selection committee members). Additionally, for more on leadership, the selection committee members’ page has a link to a SSHRC-funded study on leadership at the graduate studies and postdoctoral levels.

Vanier CGS nominees are assessed on three criteria – academic excellence, research potential and leadership (potential and demonstrated ability) with each being scored between 0.1 and 9.0 (in increments of 0.1, with 9.0 being the highest and 0.1 being lowest).Footnote 26 Academic excellence is defined in terms of the candidate’s research history and the impact of their activities to date in their area(s) of expertise and in the communities associated with their research while research potential is defined as that demonstrated by the candidate’s research history, their interest in discovery, the proposed research and its potential contribution to the advancement of knowledge in the field and any anticipated outcomes.

Leadership was the only criterion that the program recognized as needing to be assessed in an indirect manner given that there is no opportunity to interview candidates and provides a list of indicators and examples of their sources. The indicators are: personal achievement, involvement in academic life, volunteerism/community outreach, civic engagement, goal achievement, self-management, integrity, “other characteristics” and social skills. The indicator labelled “other characteristics” has as example sources, assessing whether the candidate “is creative and takes initiative; is curious; deals well with complexity; has a strong sense of reality; is courageous; is strategic; a big-picture thinker; focuses on solutions, not problems; is capable of producing extraordinary results; and is able to solve real problems and create real products.”

The inclusion of leadership as a selection criterion was seen as one of the factors that distinguishes the Vanier CGS program from other federal government doctoral scholarship programs; however, the use of this selection criterion remains a challenge. All respondents interviewed from the Tri-agencies and the institutions (selection committee members, scholarship liaison officers and academic supervisors) described the criterion as the most subjective of all the criteria. Consistent with the 2014 evaluation findings, interviewees reported that it was particularly challenging to assess the leadership of foreign applicants due to international variability in the definition and operationalization of leadership, and access to leadership opportunities. Specifically, the majority of selection committee members (6/7) reported difficulties operationalizing the concept at review committee sessions given its subjectivity.

Of the three selection criteria, scores for the leadership component were the lowest and most variable. During the period under review (2013-14 to 2017-18), the mean scores for the different criteria were (in descending order of magnitude): academic excellence, 5.84 (SD = 1.83, n = 2623); research potential, 5.16 (SD = 1.90); and leadership, 4.80 (SD = 1.98). The evaluation also found that applications from foreign citizens (M = 4.60, SD = 1.96, n = 776) were scored significantly lower (p < .01) on the leadership criterion than those from Canadians and permanent residents (M = 4.88, SD = 1.98, n = 1847). Foreign citizens were also rated significantly lower than Canadians and permanent residents on academic excellence (M = 5.64, SD = 1.76, n = 776 vs. M = 5.92, SD = 1.85, n = 1847; respectively) and final scores (M = 5.13, SD = 1.57, n = 776, vs. M = 5.32, SD = 1.58, n = 1847; respectively); although the mean scores were not practically different.

These findings demonstrate that in spite of the steps taken by the VBS in response to the previous evaluation’s findings to improve the assessment of leadership, challenges still remain.

The Vanier CGS program is being delivered in a cost efficient manner

Evaluation findings indicate that the Tri-agencies are delivering the Vanier CGS program in a cost efficient manner. The ratio of direct program administrative costs to total program expenditures and the proportion of a program’s budget that is expended both speak to how efficiently a program is being run. The evaluation found the ratio of direct administrative costsFootnote 27 to total program expenditures to be low, remaining between 1.7% and 1.9% since 2014-15 (Figure 6: Vanier CGS administrative costs as a percentage of total program expenditures). The previous evaluation reported a decline in the ratio from 8.7% to 2.2% between 2009-10 and 2013-14.

All senior Tri-agency officials interviewed (5/5) corroborated the finding that the program was being delivered efficiently, with several of them noting that the VBS model provides cost-efficient delivery. Instead of each Tri-agency dealing with its own portion of the program, the VBS was set up in October 2012 to take care of the day to day administration of the program including providing administrative support to the selection committees. The secretariat is located at CIHR and comprises an executive director, manger and program delivery staff from each of the Tri-agencies.

Performance

Key Findings

The Vanier CGS is not effectively attracting international students to Canada and retaining domestic students in Canada at time of application

As established in the design and delivery section, the current design of the Vanier CGS program means that it is not effectively meeting its key immediate expected results to attract and retain top doctoral trainees at time of application. The number of foreign trainees applying from outside Canada was low (less than 2%) and the award did not appear to play a role in trainees’ decisions to come to or remain in Canada because the majority of recipients reported that they were already enrolled in a doctoral program (8%) or would have enrolled in the same doctoral program had they not received the award (80%).

National and international awareness of Vanier CGS has increased

The evaluation found that national and international awareness of the Vanier CGS had increased over the period under consideration with the program’s website and social media pages registering increased activity. The Vanier CGS program has been promoted in Canada and around the world through a variety of methods including institutional websites, the Vanier CGS website, social media (Facebook, Twitter, LinkedIn), media relations, Global Affairs Canada’s EduCanada promotional activities, and special events (e.g., conferences, trade shows and information sessions on scholarships organized by Canada’s missions abroad). Web metrics data from the VBS show that visits to the Vanier website doubled between 2014 and 2017, the period for which data is available (Figure 7: Vanier CGS website traffic and sources, 2014-2017). During the same period, the proportion of traffic originating from Canada dropped from 37% to 24% whereas traffic from international sources increase from 63% to 76%.

The Vanier program has a Facebook page on which the profiles of recipients in university newspapers are usually posted thereby drawing considerable attention to the program over the years. In 2014, the Vanier Facebook page had 6,518 fans and between January 2017 and January 2018, the Facebook page added 11,200 new followers. The increased activity on the program’s website and Facebook page suggest an increase in awareness of the program on the part of the general public and potentially among students considering doctoral studies in Canada.

The evaluation found that unlike the general public, Vanier CGS recipients’ use of the program’s social media or other outlets was minimal. Results from the tracer survey showed that most recipients (n = 261) were not using the Vanier CGS Facebook group (69%), LinkedIn group (68%), or electronic newsletter (67%) to keep informed of Vanier CGS related updates and activities. Additionally, between 15% and 20% were not aware of these options for keeping informed about the Vanier CGS. Corroborating the survey findings, end of award reports showed that less than half of the 348 respondents were aware of the Vanier CGS LinkedIn group (41%) or had registered as members (36%) and the majority of registered members of the group felt it was not very or not at all useful (70% of 125). These results are consistent with the 2014 evaluation findings that the most common ways that recipients learned about the program was through interpersonal communication with colleagues, friends, graduate program coordinators or doctoral supervisors.

Vanier CGS recipients (n = 348) indicated through end of award reports, that the top four reasons they applied for the scholarship were the high financial value, prestige, perceived ability to increase recipients’ potential to obtain their desired employment in future, and opportunity to develop their research leadership potential. There were some differences why trainees applied for the award, based on citizenship status. Specifically, foreign citizens were more likely than Canadians and permanent residents to see the Vanier CGS as offering them an international training opportunity (p < .01) or as being essential for their desired area of employment (p < .05). Although it should be noted that both of these factors were amongst the least frequent reasons why recipients applied for the Vanier CGS (Figure 8: Reasons for applying for the Vanier CGS by citizenship status).

Vanier recipients are perceived to be top-tier trainees

Consistent with the 2014 evaluation results, analysis of administrative data showed that although the applicant pool is limited by the design, competition to obtain a Vanier CGS award was very strong and therefore selected candidates appear to be top-tier doctoral trainees. As previously noted in the Relevance section of this report, over the review period, out of an average of 1,084 candidates who applied annually through the eligible Canadian institutions, only about 15% were eventually awarded the scholarship (31% of those nominated), while about 51% did not even make it past their respective institutions’ internal selection processes. Another indication of the quality of recipients is that by the time they received notification of the Vanier CGS award, over half of end of award report respondents (65%, n = 348) had already received other offers of doctoral support.

Academic supervisors considered the recipients to be exceptional, describing them as “world-class” or “top-tier” scholars and among the highest quality students with whom they had ever worked. To further emphasize the quality of recipients, approximately three-quarters of selection committee members (5/7) spoke of a significant difference between Vanier CGS recipients and recipients of CGS-D awards, noting that although both groups were academically exceptional, the key distinguishing factor was the leadership criterion with the Vanier scholars having the potential to be strong leaders. It should be noted however, that other selection committee members (2/7) felt that differences between Vanier scholars and recipients of other federal level doctoral awards (e.g., CGS-D) were minimal, and that there is substantial overlap between the two award programs.

Recipients are most involved and show greatest improvement in the areas of research and professional leadership

The level of involvement and improvement in various skills related to research, teaching, service and professional development varied for Vanier CGS recipients, consistent with the previous evaluation (2014). Recipients (n = 348) were asked to indicate the extent to which they felt their leadership abilities developed through the award via end of award reports. Overall, the extent varied from moderate to great across the four leadership areas. Specifically, the development of research leadership was the greatest (M = 4.09 out of 5, SD = 0.83) followed by personal/professional leadership (M = 3.85, SD = 0.88), while the development of service leadership (M = 3.39, SD = 1.10), and teaching leadership (M = 3.21, SD = 1.18) was moderate.

Recipients were also asked to indicate the extent of involvement and improvement in specific activities within each of the four leadership areas (Figure 9: Extent of involvement in and improvement of research leadership activities during Vanier CGS). In terms of research leadership activities, recipients reported greatest involvement in development of research ideas/questions, data collection and interpretation of research findings. While involvement in the other three leadership areas was limited, critical and creative thinking (professional leadership), organizing or participating in volunteer activities (service leadership), and communication/presentation activities (teaching leadership) were the activities recipients indicated greatest involvement in. Improvement generally mirrored involvement across all four leadership areas, with individuals indicating greater improvement in activities they were more involved in.

Figure 9: Extent of involvement in and improvement of research leadership activities during Vanier CGS

Figure 9 – Long description
Involvement Improvement
Interpretation of research findings 4.40 4.05
Development of research ideas/questions 4.27 4.01
Knowledge translation/mobilization 4.23 3.97
Data collection phase/research implementation 4.28 3.91
Development of research protocols/methods 4.03 3.8
Preparing research/funding proposals 3.43 3.50
Knowledge of research integrity/ethical conduct 3.72 3.44
Multidisciplinary/interdisciplinary research 3.62 3.43
Leading research projects (human, financial, and time management 3.38 3.26
International research collaborations 3.19 3.18
Collaborative research with the private sector/government/not-for-profit 2.51 2.52

Source: Vanier CGS End of Award Report (VEAR) as of February, 2019, analyzed by the Evaluation Unit. n = 348.

The extent of improvement across a range of skills in the four leadership areas also varied on average from slight to great, with the greatest improvement observed for research related skills followed by professional development skills (ranging from moderate to great on a 5 point scale; Figure 10: Extent of involvement and improvement of professional leadership activities during Vanier CGS), while service related and teaching related skills ranged from slight to moderate (See Figure 11: Extent of involvement and improvement of service leadership activities during Vanier CGS and Figure 12: Extent of involvement and improvement of teaching leadership activities during Vanier CGS). These findings were consistent with those from the 2014 evaluation.

Interview findings corroborated Vanier scholars’ involvement in and demonstration of leadership, not only in research but in other areas. Over half of the Vanier CGS recipients interviewed (8/14) indicated that they engaged in leadership activities during their doctoral degree for example, participating on graduate committees and graduate student initiatives, hiring and leading research teams, co-chairing committees, organizing round tables in university departments, teaching, presenting, and leading discussions. Half of the recipients (7/14) specifically identified that the financial support from the Vanier CGS allowed them the time and resources to engage in leadership activities such as engaging in unpaid volunteer work or projects, and not having to be a teaching assistant or take on other employment.

Recipients demonstrated research productivity during their award

Consistent with the previous evaluation results, end of award reports show that Vanier recipients (n = 348) were generating publications, conference presentations and other knowledge outputs during the tenure of the scholarship. The majority of recipients (81%) had at least one peer-reviewed journal article published or in press. The average number of published or in-press peer reviewed publications was 3.92 (SD = 2.31, n = 283), which is lower than that reported in the 2014 evaluation (M = 4.5, SD = 6.0). There were significant differences by agency. Specifically, both NSERC and CIHR had higher average numbers of publications compared to SSHRC (p < .001).Footnote 28 Few recipients (12-24%) were involved in publishing other written materials, including grey literature, reports and/or non-peer reviewed articles, a finding that is consistent with the previous evaluation.Footnote 29

Vanier recipients attended local, national and international conferences (including both invited and uninvited presentations) during the tenure of their award. Over half (57%) attended at least one local conference and at least one international conference (55%), while almost half attended at least one national conference (42%). The average number of local presentations ranged from 2.68-2.90, while the average number of national presentations ranged from 2.05-2.68 and international from 2.73-3.95. The average number of presentations was much lower than that reported in the 2014 evaluation (national - M = 9.2; international - M = 4.5). There were significant differences by agency. Specifically, both CIHR and SSHRC respondents had more local invited publications (p < .001), whereas NSERC and CIHR had more local uninvited presentations (p < .01). Lastly, NSERC respondents had more international presentations (p < .05).

One-third of end of award report respondents (33% out of 345) applied for research grants during the tenure of their award and 28% received at least one research grant (88% of the 113 who applied). Across agencies, more CIHR-funded recipients (40% of 110) than NSERC (25% of 114) or SSHRC-funded recipients (29% of 111) applied for at least one research grant during the tenure of their Vanier CGS and up to a third received at least one grant (CIHR – 33%, SSHRC – 27% and NSERC – 23%).

All academic supervisors interviewed (5/5) confirmed that the Vanier recipients they supervised conducted independent research and believed that receiving the Vanier CGS award enabled these scholars to conduct research in an area of their own interest, which was in some way linked to the supervisor’s field of study. Further, all supervisors (5/5) felt the Vanier scholars demonstrated a high-degree of initiative in conducting research and were perceived as leaders in the academic community.

Recipients and applicants have established research collaborations

Vanier recipients established national and international research collaborations during their award, many of which continued on after the award. There were minimal differences between recipients and applicants in terms of the types and frequency of collaborations after completing their doctoral degree. Consistent with the 2014 evaluation findings, there was more collaboration within than across disciplines and more collaboration within than outside Canada.

End of award reports showed that in addition to their supervisors and mentors, recipients (n = 348) also actively collaborated with other researchers (45-91%) and trainees (39-87%) within and outside their disciplines as well as within and outside Canada (Figure 13: Recipient's collaboration with researchers and trainees during the Vanier CGS). Three-quarters (75%) of Vanier recipients (n = 262) stated that they established at least one formal collaboration during their award and the majority of these recipients (88%) indicated they planned to continue with the collaboration(s) after completion of the award.

Figure 13: Recipient's collaboration with researchers and trainees during the Vanier CGS
Interacted more than once a year
Researchers in your discipline, inside Canada 90.2%
Trainees in your discipline, inside Canada 87.4%
Researchers in other disciplines, inside Canada 70.5%
Trainees in other disciplines, inside Canada 70.1%
Researchers in your discipline, outside Canada 68.9%
Trainees in your discipline, outside Canada 58.6%
Researchers in other disciplines, outside Canada 44.6%
Trainees in other disciplines, outside Canada 38.9%

Source: Vanier CGS End of Award Report (VEAR) as of February, 2019, analyzed by the Evaluation Unit. n = 348.

The views of Vanier supervisors interviewed are consistent with the end of award report findings as all supervisors (5/5) stated that the Vanier recipients they supervised had established national and international collaborations during their award and that these collaborations continued beyond the tenure of their awards.

Findings from the tracer survey confirmed those from the end of award reports but also indicated that the frequency and breadth of collaborations decreased after the award. Both Vanier recipients and applicants (n = 468) reported collaborating more often (interacting at least once a week) with researchers within their discipline both nationally and internationally (no significant differences between the groups) and less often with researchers outside of their discipline both nationally (recipients 29% vs. applicants 21%, n = 468, p < .01) and internationally (no significant difference between the groups).Footnote 30

The collaborations resulted in the creation of knowledge products such as publications and presentations. Compared to applicants (n = 271), Vanier recipients (n = 162) reported significantly more publications (M = 6.05, SD = 13.24 vs. 4.09, SD = 7.42, p < .05) and presentations (M = 6.61, SD = 18.96 vs. 3.83, SD = 7.59, p < .05) with researchers within their discipline in Canada. On the other hand, there were no differences between recipients (n = 150) and applicants (n = 256) in the number of outputs with researchers outside Canada with regard to publications (M = 3.45, SD = 8.46, n = 150 vs. M = 3.49, SD = 7.91, n = 256) or presentations (M = 3.32, SD = 9.86 vs. M = 2.92, SD = 6.94). CIHR respondents reported more publications with researchers in Canada than SSHRC respondents and NSERC respondents reported more publications with researchers outside Canada than SSHRC respondents. These differences are likely due to differing publication patterns between researcher across the domains of health, natural sciences and engineering, and social sciences and humanities. Male respondents reported more publications with researchers outside Canada than females.

Collaborations with researchers in other disciplines (within and outside Canada) resulted in very few publications and conference presentations with no differences between recipients and applicants (M = 0.89-1.36, SD = 3.37-5.78, n = 128-148 recipients and 211-217 applicants).

Majority of both Vanier recipients and applicants worked full time in the academic sector in Canada

Consistent with previous evaluation results, majority of the tracer survey respondents (n = 508) were in full time positions in Canada (73%), mainly in the academic sector (59%) with no observable differences between Vanier recipients and applicants. Overall, the majority of respondents (81%) were working full time (30 hours or more per week) with very few working part time (5%), self employed (4%) or unemployed (3%). The majority of those currently employed (n = 460) were living in Canada (72%). By comparison, the previous evaluation reported that 63% of Vanier scholar graduates (out of 103) were living in Canada post-graduation. Retention levels of trainees employed in Canada post-graduation is greater for Canadian citizens; however, half of foreign citizens and two-thirds of permanent citizens remain in Canada to work post-graduation. Specifically, significantly more Vanier CGS recipients who were Canadian citizens were working in Canada (80%) compared to permanent residents (66%) and foreign citizens (49%).

Approximately two-thirds of recipients and applicants (n = 460) work in the academic sector (59%) with the remainder in the private sector or industry (14%), the health care sector (12%), the public sector (8%), and the non-profit sector (6%) (Figure 14: Employment sector by funding status (Vanier CGS recipients and applicants)).Footnote 31 There were sex and agency differences between recipients and applicants with regard to sector of employment (p < .05, p < .01; respectively).Footnote 32 Specifically, more females were employed in the academic sector while more males were employed in the private sector. More SSHRC respondents were employed in the academic sector compared to CIHR and NSERC.

Figure 14: Employment sector by funding status (Vanier CGS recipients and applicants)

Figure 14 – Long description
  Public sector/ Government Non-profit organization Hospitals and other health care providers Private sector/Industry Academic Other Don't know/prefer not to answer Total
CIHR Applicants 4% 5% 30% 9% 53% 0% 0% 73
CIHR Recipients 8% 8% 26% 7% 52% 0% 0% 90
NSERC Applicants 9% 1% 6% 33% 51% 0% 0% 73
NSERC Recipients 8% 3% 6% 22% 59% 2% 0% 64
SSHRC Applicants 10% 10% 0% 5% 70% 2% 2% 76
SSHRC Recipients 8% 8% 1% 4% 71% 6% 1% 84
Male Applicants 8% 4% 13% 20% 55% 0% 0% 96
Male Recipients 8% 6% 15% 16% 53% 2% 0% 100
Female Applicants 8% 6% 10% 12% 61% 1% 1% 126
Female Recipients 8% 7% 9% 5% 67% 3% 1% 138
Applicants (n = 222) 8% 5% 11% 16% 58% 1% 1% 222
Recipients (n = 238) 8% 7% 12% 10% 61% 3% 0% 238

Source: Survey of Vanier CGS recipients and applicants, 2019.

Overall, respondents rated their careers as research intensive (M = 3.78 out of 5, SD = 1.30, n = 460) and believed their doctoral training had influenced to a great extent their decision to pursue a research intensive career.Footnote 33 There were sex and agency differences in the reported research intensity of careers (p < .05 for both).Footnote 34 Specifically, males rated their careers as more research intensive than females, although the difference in practical terms is quite small. CIHR respondents rated their careers as more research intensive than SSHRC respondents.

Recipients and applicants interviewed (n = 26) felt their doctoral training gave them the requisite skills and confidence to undertake research careers in Canada. All supervisors (5/5) believed that the recipients they supervised had gone on to undertake research careers either nationally or internationally.

Vanier recipients were more likely to be assistant or associate professors, but there was no difference between recipients and applicants in tenure status

Findings from the tracer survey showed that among those working in the academic sector, Vanier CGS recipients were significantly more likely to be in more advanced academic positions (associate professor, assistant professor or research faculty/scientist/fellow) while applicants were more likely to be in transitional academic positions (postdoctoral fellow or instructor) (Figure 15: University position by funding status (Vanier CGS recipients and applicants) – Academic sector only).Footnote 35 The previous evaluation found that 51% of recipients who had graduated and were employed in the academic sector (n = 65), were working as postdoctoral fellows/associates (compared to 22% in the current evaluation; although, it should be noted that this figure was expected to decrease as the same cohorts were surveyed in the first evaluation). There were differences in academic positions held by agency (p < .01) and sex (p < .05).Footnote 36 Specifically, more SSHRC respondents, and more males, were in assistant professor positions.

Figure 15: University position by funding status (Vanier CGS recipients and applicants) – Academic sector only

Figure 15 – Long description
  Other Chair, Dean (any level), or Department Head Staff Scientist Full Professor Associate Professor Assistant Professor Instructor Adjunct Professor Research Faculty, Scientist, Associate, or Fellow Postdoctoral fellow Research Assistant Total
CIHR Applicants 11% 0% 5% 0% 0% 23% 6% 2% 7% 46% 0% 39
CIHR Recipients 6% 2% 4% 0% 11% 32% 2% 2% 13% 28% 0% 47
NSERC Applicants 2% 0% 2% 2% 10% 36% 5% 0% 0% 42% 0% 38
NSERC Recipients 8% 0% 0% 3% 8% 34% 8% 0% 13% 26% 0% 38
SSHRC Applicants 13% 0% 2% 0% 3% 43% 14% 3% 6% 14% 2% 52
SSHRC Recipients 7% 0% 0% 0% 15% 50% 7% 2% 5% 15% 0% 60
Male Applicants 2% 0% 3% 2% 5% 46% 8% 0% 3% 32% 0% 53
Male Recipients 2% 0% 0% 2% 17% 38% 4% 0% 17% 21% 0% 53
Female Applicants 15% 0% 2% 0% 4% 28% 10% 3% 6% 32% 1% 76
Female Recipients 10% 1% 2% 0% 9% 41% 7% 2% 5% 23% 0% 92
Applicants (n = 129) 9% 0% 3% 1% 4% 35% 9% 2% 5% 32% 1% 129
Recipients (n = 145) 7% 1% 1% 1% 12% 40% 6% 1% 10% 22% 0% 145

Source: Survey of Vanier CGS recipients and applicants, 2019.

Among those employed in the non-academic sectors, recipients were more likely to be in junior or executive level positions while applicants were more likely to be in senior or management level positions. (Figure 16: Position level by funding status (Vanier CGS recipients and applicants) – Non-academic sectors). There were differences by agency (p < .05).Footnote 37 Specifically, more SSHRC respondents were in executive level positions.

Findings from the tracer survey showed no significant differences between Vanier recipients and applicants regarding their tenure status. Among those working at an instructor level or higher in the academic sector (n = 148), the majority of recipients and applicants (66%) were in tenure-track positions while 17% were in non-tenure track positions and 16% had obtained tenure. There were however, overall differences with respect to agency (p < .05) and sex (p < .01).Footnote 38 Specifically, more CIHR respondents were in tenure track position while more NSERC respondents were already tenured. More male respondents were in tenured or tenure track positions.

On average, it took 2.26 years after completing their doctoral degree (SD = 1.81) for recipients and applicants (n = 99) to obtain a tenure track position and 4.06 years (SD = 2.02) to obtain tenure. There were agency differences in time to obtain a tenure track position (p < .05).Footnote 39 Specifically, SSHRC respondents took less time to obtain a tenure track position compared to CIHR and NSERC.

Recipients had higher employment-related income, but no difference with applicants in job satisfaction

The tracer survey results showed that, on average, Vanier CGS recipients (M = $98,984, SD = $71,739.7) reported a higher annual employment-related income than applicants (M = $85,168, SD = $49,545.2, p < .05) and there were agency and sex differences. SSHRC respondents (M = $79,932, SD = 34,963.4) earned substantially less than CIHR (M = $99,223, SD = 76,608.9; p < .05) but not NSERC (M = $91,287, SD = 55,797.3) respondents, and males (M = $100,139, SD = 77,376.5) earned more than females (M = $82,577, SD = 38,014.7; p < .05).Footnote 40

Overall, Vanier recipients and applicants (n = 460) were very satisfied with their current employment (M = 4.24 out of 5, SD = 0.92) and strongly believed that their current job was related to their doctoral studies (M = 4.15, SD = 1.17). However, more Vanier recipients (85% of 236) than applicants (72% of 218) indicated that their current job was aligned with their career development goals (p < .01).

Vanier CGS and other awards had positive impacts on research careers

The tracer survey findings further demonstrate that the Vanier CGS (for recipients) and their doctoral training (for applicants) had an impact on their careers. Respondents who described their career as research intensive indicated that the Vanier CGS (M = 3.61, SD = 1.34, n = 146) or their doctoral training (M = 4.20, SD = 1.10, n = 260) had greatly influenced their decision to undertake a research career. The end of award reports showed that almost all recipients employed in the academic sector (98%, n = 90) believed that the Vanier award was an important factor in their decision to remain in an academic research environment.

Additionally, end of award reports showed the importance of the Vanier CGS experience to recipients (n = 348). The majority felt that their supervisor gave them adequate guidance and feedback on their research (86%) and encouraged them to pursue a career in research (78%). The majority also felt that the experience gained during their Vanier CGS increased their desire to pursue a career in research (80%) and that it would improve their chances of getting a permanent job in a relevant area (89%).

These results were corroborated by interviewees. Half of the recipients interviewed (5/10) noted that the Vanier CGS provided an advantage when applying for jobs or post-doctoral awards. For example, interviewees noted: “It's [Vanier CGS] a prestigious award” and “The CV is more competitive.” In addition, almost three quarters (7/10) indicated that the Vanier CGS allowed for time and resources to enhance leadership skills, which supported their ability to obtain work and their current position.

Similarly, half of applicants interviewed (6/12) indicated that the training they received in their doctorate and post-doctorate degrees (e.g., regarding data collection, data privacy, research management) was important for their career development, as were other awards they received (e.g., awards facilitated advancement in research career through recognition and networking).

Half of the recipients (5/10) indicated that the Vanier CGS provided financial, academic and intellectual freedom to conduct research, which helped them realize the importance of research and build confidence as a researcher. Some recipients and applicants (4/14) said that they would have pursued a research career without the Vanier CGS and/or had already decided to pursue a research career prior to receiving the award.

Just over half of applicants interviewed (7/12) felt that the result of the Vanier CGS competition did not impact their career decisions or other opportunities, while a few (2/12) felt they had fewer opportunities during their doctoral studies as a result of not receiving the Vanier CGS (e.g., could not attend important international conferences or had fewer career development opportunities).

Recipients continued to show greater research productivity post-award

The Vanier CGS program considers three equally weighted selection criteria - academic excellence, research potential and leadership - and the leadership component is seen as the main element distinguishing it from other sources of support for doctoral training in Canada. In line with this distinction, a key expected outcome of the Vanier CGS program is for recipients to become leaders as the next generation of researchers in Canada. The evaluation assessed the extent to which this has occurred post degree completion among scholarship recipients in comparison with applicants in terms of their productivity (publications and conference presentations), success at obtaining grants and awards and their perceptions of the impact of leadership development opportunities on their careers.

Findings from the tracer survey confirm that the research productivity shown by recipients during the tenure of the Vanier CGS award continued after degree completion as they generated more peer reviewed publications and international conference presentations than applicants.

Vanier recipients produced significantly more first author peer reviewed publications compared to applicants (M = 5.14, SD = 5.86, n = 294 vs. M = 3.28, SD = 4.24, n = 166; p < .05), with Vanier recipients attributing an average of 2.44 (SD = 2.87) of these publications to the Vanier CGS.

CIHR respondents (M = 5.03, SD = 6.80) reported more publications as first author than NSERC (M = 3.65, SD = 3.85) and SSHRC (M = 3.12, SD = 3.10, p < .05) respondents and men (M = 4.61, SD = 5.70) reported more first author publications than females (M = 3.47, SD = 4.28; p < .05). CIHR respondents (M = 7.82, SD = 14.11) and NSERC respondents (M = 5.33, SD = 7.32) reported more peer-reviewed publications not as first author than SSHRC respondents (M = 1.76, SD = 3.91, p < .05); and male respondents (M = 6.40, SD = 10.37) reported more of those than females (M = 3.98, SD = 9.39, p < .05).

Recipients (M = 8.16, SD = 11.81, n = 166) had significantly more international conference presentations than applicants (M = 5.29, SD = 7.84, n = 294; p < .05) and attributed an average of 2.6 to the Vanier CGS (SD = 4.67). Recipients also had more presentations at national conferences (M = 6.73, SD = 10.52) compared to applicants (M = 4.60, SD = 11.66) and attributed 2.11 (SD = 3.80) to the Vanier CGS; however, the difference was not statistically significant.

There were no differences between recipients and applicants in funding received following their doctoral degrees

Overall, the tracer survey findings showed that just under two-thirds of Vanier CGS recipients and applicants (61% out of 508) received funding since completing their doctoral studies (with no differences between the groups). This funding mainly included postdoctoral awards (72%), research grants (54%), and other awards or grants (11%). The funding was mainly from the Tri-agencies, international organizations, Canadian universities and provincial governments with no observable differences between recipients and applicants.

In addition, there were no observable differences in the type of grant or award received by recipients and applicants who reported receiving Tri-agency funding after completing their degree (n = 178). The specific awards or grants received most frequently included the CIHR Fellowship (24%), the Banting Postdoctoral Fellowship (17%), the NSERC Postdoctoral Fellowship (13%), the SSHRC Insight Development Grant (11%), and the CIHR Project Grant (10%).

Few differences between recipients and applicants on the impact of leadership development opportunities

The tracer survey findings confirm that both recipients and applicants (n = 508) generally believed that the diverse leadership development opportunities they experienced during the tenure of their scholarship or doctoral training had positive impacts on their career development. The most common opportunities received during their doctoral degrees were research skills and training experiences (95%), teaching experiences or skills (75%), obtaining other funding (68%), generating publications (67%), mentorship (52%), professional experiences (50%), international experiences (46%), and leadership experience (44%). There were some observable differences between recipients and applicants. Specifically, recipients were more likely than applicants to indicate that obtaining funding (74% vs. 65%), leadership experience (54% vs. 39%), and international experiences (51% vs. 44%) acquired during the tenure of their scholarship or doctoral training had a positive impact on their careers. However, applicants were more likely to indicate that teaching experiences or skills (79% vs. 69%) had a positive impact on their careers.

Overall, both recipients and applicants believed that the leadership development opportunities they experienced during their degree prepared them for their career to a great extent, with no significant differences between the groups (M = 3.96 out of 5, SD = 0.95 vs. M = 3.87, SD = 0.95; respectively).

Conclusions and Recommendations

Conclusions

Relevance

There is a continued need to support top doctoral students

The evaluation concludes that there is a continued need to support top doctoral students (both domestic and international) in Canadian institutions based on GOC priorities and uptake from trainees and institutions. Canadian universities have also seen steadily increasing international student enrollment rates, with international students representing 14% of total enrollments in 2017-18.

The Vanier CGS program’s expected results of attracting and retaining top doctoral students and supporting the next generation of researchers address key priorities outlined in recent GOC documents - Budget 2018, Budget 2019, Canada’s Science Vision and Canada’s Fundamental Science Review. The Vanier CGS program’s expected outcomes (attraction and retention of top doctoral students) and key design features (open to foreign citizens, leadership as key selection criterion, and highest per annum award amount) means that it occupies a niche among the federal government doctoral support programs. The Vanier CGS program is aligned with the GOC and the tri-agencies roles and priorities as outlined in the Acts and research capacity building priorities of the federal granting agencies.

There was clear uptake of the Vanier CGS program from both trainees and institutions. Eligible institutions received an average of 1,084 applications annually for the period 2013-14 to 2016-17 of which, an average of 49% were nominated. Institutions fulfilled 91% of their available nominations quota during the 2015-2018 allocation cycle.

Despite the specific nature of the program’s design, alignment with priorities and clear uptake, the ongoing challenges encountered by the program to effectively achieve its objectives means that it is not clear that the Vanier CGS program, as currently designed and delivered, is an effective means to fulfill the need to support top doctoral students (both domestic and international).

Design and Delivery

The Vanier CGS Program’s current design limits the attraction and retention of top doctoral students at time of application

The design and delivery of the Vanier CGS program is limiting the achievement of two key expected results, outlined in the program authorities, to attract to Canada top students from other countries and retain Canada’s top doctoral students at time of application. The program’s delivery model limits both the population of potential applicants to the 57 eligible Canadian institutions and the nomination and funding of foreign students from outside of Canada. More specifically, applicants can only be nominated by one of 57 eligible Canadian institutions that have been allocated a quota from at least one of the federal granting councils. Furthermore, the evaluation found that almost all recipients were already enrolled at eligible institutions at the time of application: 61% of applicants were Canadian students and 37% were foreign candidates living in Canada. During the period under review, only 1.5% of applications were from candidates applying from outside of Canada. Thus, the evaluation concludes that the Vanier CGS program, as currently designed and delivered, does not support the achievement of intended outcomes.

The evaluation found that assessing the leadership of applicants remains a challenge due to subjectivity causing difficulty in interpretation of and inconsistency in the assessment of the leadership criterion. Of the three selection criteria, scores for the leadership criterion were the lowest and most variable. Foreign applications were scored significantly lower on the leadership criterion than Canadian citizens and permanent residents. Similarly, foreign applications were also scored lower on academic excellence and final scores.

In terms of EDI factors, overall no barriers were reported related to Indigenous status, being a visible minority, or a person with a disability. With respect to gender, women were significantly more likely to experience barriers related to gender than men. Given the small sample sizes associated with some of these categories, as well as the findings related to the assessment of leadership and selection of international students, further study is needed to assess EDI factors overall. This work will be important to help ensure the equitable assessment and selection of Vanier recipients and support commitments made by CIHR, NSERC and SSHRC in the Tri-Agency Statement on Equity, Diversity and Inclusion.

The evaluation found that the federal granting agencies were delivering the Vanier CGS program in a cost efficient manner as evidenced by a low percentage of direct administrative costs to total program expenditures (between 1.7% and 1.9% for the period under review).

Performance

The Program is not effectively attracting top international students and retaining top Canadian doctoral students

The evaluation found that the Vanier CGS program is not achieving key immediate expected outcomes. The program is not effectively attracting international students to Canada as almost all international students were already living in Canada at the time of application. The program was also not effectively retaining domestic students in Canada at the time of application as the majority of recipients were already enrolled in a doctoral program in Canada or would have enrolled in the same doctoral program had they not received the award.

The evaluation showed that national and international awareness of the Vanier CGS program had increased. The program was known for its high award value, prestige, perceived ability to increase recipients’ potential to obtain their desired employment in future, and the opportunity it afforded them to develop their research leadership potential.

Vanier recipients are considered to be top-tier by their academic supervisors and the majority of selection committee members interviewed. The evaluation demonstrated that Vanier recipients were engaging in research, teaching and service leadership opportunities, with the greatest development in the area of research leadership. Vanier recipients were establishing national and international collaborations during the award, many of which continued after the award. However, there were minimal differences between Vanier recipients and applicants in terms of the type and frequency of collaborations following their doctoral degree.

Majority of both recipients and applicants live in Canada and work in the academic sector and were in research intensive careers

Although the Vanier CGS program is achieving its expected intermediate outcomes, the evaluation found that applicants (who did not receive the award) report very similar outcomes, which calls into question the incremental value of the program. The evaluation found that majority of both Vanier recipients and applicants were living and working in Canada, working in the academic sector, and were in research intensive careers. Among Vanier recipients, Canadian citizens were more likely to be working in Canada (80%) than permanent residents (66%) and foreign citizens (49%). Vanier recipients were more likely to be in more advanced academic positions (e.g., assistant or associate professor) compared to applicants (e.g., postdoctoral fellow), but there were no differences in tenure status and time to tenure. Overall, both recipients and applicants had a high degree of job satisfaction, while Vanier recipients had a higher average income than applicants.

The evaluation showed that Vanier recipients had higher research productivity (as measured by peer reviewed publications and conference presentations) than applicants; however, there were only minimal differences between the two groups regarding additional awards and the impact of leadership development opportunities. There were also little to no differences in their degree experiences, impact of leadership development opportunities, and additional funding post degree.

Given that key design and delivery aspects continue to limit the achievement of intended outcomes, especially the attraction and retention of top doctoral students at time of application, there is a need to change the design and delivery of the program to better achieve the current objectives and/or revise the objectives to better align with the current delivery model. Further, the evidence indicating the limited incremental impact of the program on recipients when compared to applicants in key outcome areas reinforces the need to reconsider the program’s design, delivery and objectives in relation to the CGS-D program as well as agency-specific doctoral scholarship programs.

Recommendations

The evaluation makes the following recommendations aimed at improving the ongoing implementation and performance of the program.

Recommendation 1:

The Vanier CGS program needs to change its current objectives and/or design and delivery model in order to more effectively attract top doctoral students to Canada and retain top domestic students in Canada at time application. Given the similar levels of achievement by recipients and applicants related to leadership opportunities, post-graduation retention and career outcomes, the changes to the Vanier CGS program need to be made with due consideration of other federal doctoral scholarship programs.

Recommendation 2:

In light of the evaluation findings and the Tri-Agency Statement on Equity, Diversity and Inclusion, the Vanier CGS program should examine the nature and extent of EDI barriers, including GBA+ analysis, related to the potential biases associated with the leadership selection criteria, the nomination process, and the selection of international doctoral students in order to more equitably assess and select Vanier recipients.

Appendix A – Tables and Figures

Figure 1: Vanier CGS Logic Model

Vision:To attract and retain top-tier doctoral talent, both nationally and internationally, to develop their leadership potential and to position them for success as research leaders of tomorrow, positively contributing to Canada’s economic, social and research-based growth through a research-intensive career.

Activities Outputs Immediate Outcomes Intermediate Outcomes Ultimate Outcomes

Program management and administration

Development and dissemination of program branding, communications and marketing strategies

Vanier-Banting Secretariat facilitates networking and collaborations

Top-tier international applicants are attracted to Canada

Vanier CGS awards are awarded to top-tier applicants

Top-tier Canadian applicants are retained in Canada

Communication/ promotion materials produced and disseminated

Vanier CGS awardees receive advanced research training in the social sciences, humanities, natural sciences, engineering or health

Vanier CGS awardees establish national and international collaborations

Increased national and international awareness of Vanier scholarships as a competitive  and recognized awards

Vanier CGS Scholars demonstrate leadership (Research Leadership, Academic Leadership and Service Leadership)

Vanier CGS alumni are retained in Canada

Vanier CGS alumni undertake research careers in CanadaFootnote 41

Vanier CGS alumni pursue advanced training in the social sciences, humanities, natural sciences, engineering or health

Vanier CGS alumni become leaders in early career researchFootnote 42

Vanier CGS program contributes to the supply of highly qualified/trained researchers for Canada’s research enterprise

Canada is a destination of choice for quality research training

Vanier CGS alumni are recognized as exemplars of Canadian research excellence

Figure 2: Comparison of doctoral scholarship programs offered by federal granting agencies

Vanier Canada Graduate Scholarship Canada Graduate Scholarship - Doctoral Canadian Institutes of Health Research - Doctoral Foreign Study Award Social Sciences and Humanities Research Council Doctoral Fellowship National Sciences and Engineering Research Council – Postgraduate Scholarship –Doctoral
(Vanier CGS) (CGS-D) (CIHR - DFSA) (SSHRC Doc) (NSERC - PGS-D)
Description/ Main Objective

To enable Canada to attract and retain world-class doctoral students by supporting Canadian and International students studying at Canadian Universities, in the social sciences and/or humanities, natural sciences and/or engineering and health.

Supports high-calibre students engaged in doctoral programs in all academic disciplines allowing them to fully concentrate on their doctoral studies to seek out the best research mentors in their chosen fields and contribute to the Canadian research ecosystem during and beyond the tenure of their awards.

To promote continued excellence in Canadian research by rewarding and retaining high-calibre doctoral students at Canadian institutions; and to foster impacts within and beyond the research environment.

Supports high-calibre students engaged in doctoral programs in all academic disciplines who are pursuing a doctoral degree in a health-related field abroad.

Expected to: Provide recognition and funding to students early in their academic research career, providing them with an opportunity to gain research experience abroad; and provide a reliable supply of highly skilled and qualified researchers.

Supports high-calibre students engaged in doctoral programs in the social sciences and humanities allowing them to fully concentrate on their doctoral studies, to seek out the best research mentors in their chosen fields and to contribute to the Canadian research ecosystem during and beyond the tenure of their awards.

Supports high-calibre students engaged in doctoral programs in the natural sciences or engineering allowing them to fully concentrate on their studies and seek out the best research mentors in their chosen fields.
Award value per annum ($)

50K

35K

35K

20K

21K
Maximum duration (years)

3

3

3

4

3
Total number of awards (at any time)

500

2500Footnote *

30

500

400
Number of (new) awards per annum

166

833

10

~2000

~1,200
Tenure

Canada

Canada

Abroad

Canada or Abroad

Canada or Abroad
Citizenship requirement

Canadian citizens, permanent residents and foreign citizens.

Canadian citizens and permanent residents.

Canadian citizens and permanent residents.

Canadian citizens and permanent residents.

Canadian citizens and permanent residents.
Key assessment criteria

Research potential, academic excellence, leadership skills

Research ability and potential, relevant experience and achievements obtained within and beyond academia

Research ability and potential, relevant experience and achievements obtained within and beyond academia

Research ability and potential, relevant experience and achievements obtained within and beyond academia

Research ability and potential, relevant experience and achievements obtained within and beyond academia

Source: Abstracted from program websites. Data on number of new awards per annum and total number of awards abstracted from Exhibit 7.1, p. 138 of Canada's Fundamental Science Review.

Footnote *

* While approximately 2,500 students held the award annually at the time of this analysis, investments made in Federal Budget 2019 will increase the total number of awards to 3,000

* referrer

Figure 6: Vanier CGS administrative costs as a percentage of total program expenditures

Fiscal Year 2014-15 2015-16 2016-17 2017-18
Total award expenditures (a) $24,700,000 $24,850,000 $24,900,000 $24,900,000
Total administrative costs (b)Footnote * $443,010 $437,501 $469,052 $487,322
Total program expenditures (c=a+b) $25,143,010 $25,287,501 $25,369,052 $25,387,322
Ratio of administrative costs to total expenditures (d=(b/c)% 1.80% 1.70% 1.80% 1.90%
Footnote *

*Includes 20% employee benefits plan and 13% accommodation costs.

Source: Award expenditure data obtained from Vanier Banting Secretariat and administrative costs data from CIHR Finance.

* referrer

Figure 7: Vanier CGS website traffic and sources, 2014-2017

Fiscal Year 2014 2015 2016 2017
Number of website visits 246,292 289,742 387,466 491,000
Web traffic from Canada 37% 31% 27% 24%
Web traffic from international sources 63% 69% 73% 76%

Source: Vanier Banting Communications Reports, 2014-15 to 2017-18. Vanier Banting Secretariat, 2018.

Figure 8: Reasons for applying for the Vanier CGS by citizenship status

Figure 8 – Long description
Total (n = 279-347) Foreign (n = 81-103) Canadian or Permanent Resident (n = 196-245)
Could not obtain desired employment 1.5 1.4 1.5
Essential for my desired area of employment* 2.6 2.9 2.5
Obtain additional research experience) 2.8 3.1 2.7
International training opportunity** 3.1 3.5 3.0
Opportunity to work with my supervisor 3.2 3.3 3.2
Opportunity to develop teaching leadership potential 3.4 3.3 3.4
Opportunity to develop service leadership potential 3.8 3.9 3.8
Encouraged to do so by professors/mentors 3.8 3.8 3.8
Supportive research or work environment 4.1 4.3 4.1
Opportunity to develop research leadership potential 4.5 4.6 4.4
Increased potential for desired position in future 4.6 4.6 4.6
Prestige of award 4.7 4.8 4.7
Financial value of award 4.9 4.9 4.9
N 348 103 245

Note: *p<0.05; **p<0.01.

Source: Vanier CGS End of Award Report (VEAR) as of February, 2019, analyzed by the Evaluation Unit.

Figure 10: Extent of involvement and improvement of professional leadership activities during Vanier CGS

Figure 10 – Long description
Involvement Improvement
Critical and creative thinking 4.31 4.12
Networking and collaboration 3.76 3.69
Interpersonal communication 3.87 3.68
Strategic planning/advice 3.82 3.65
Leading/influencing others within the academic community 3.43 3.43
Building global linkages and international networks 3.32 3.3
Leading/influencing others within the larger community 3.03 3.06
Personal/non-academic pursuits (e.g., sports, music) 3.06 2.88
Financial management 2.33 2.38
Entrepreneurship 2.27 2.32
Business management 1.96 2.01

Source: Vanier CGS End of Award Report (VEAR) as of February, 2019, analyzed by the Evaluation Unit. n = 348.

Figure 11: Extent of involvement and improvement of service leadership activities during Vanier CGS

Figure 11 – Long description
Involvement Improvement
Organizing or participating in volunteer activities 3.33 3.12
Taking on an administrative role 2.95 2.87
Educating youth or community groups 2.73 2.67
Participating in institutional associations 2.74 2.64
Taking on a leadership position with a student group (e.g., union, association) 2.72 2.6
Promoting awareness for a particular cause 2.68 2.58
Taking on a leadership position in a community organization 2.43 2.35
Fulfilling a position in a professional organization 2.28 2.21
Taking on a leadership role in campus activities (e.g., athletic, artistic or cultural) 2.25 2.21
Participating in civic initiatives 2.21 2.13
Developing and executing a fundraising campaign 1.89 1.87
Participating in campus media 1.81 1.82

Source: Vanier CGS End of Award Report (VEAR) as of February, 2019, analyzed by the Evaluation Unit. n = 348.

Figure 12: Extent of involvement and improvement of teaching leadership activities during Vanier CGS

Figure 12 – Long description
  Involvement Improvement
Communications/presentations 3.98 3.85
Supervising other students 2.98 3.14
Developing course materials 2.59 2.79
Developing teaching methods 2.45 2.66
Teaching a university course 2.39 2.6

Source: Vanier CGS End of Award Report (VEAR) as of February, 2019, analyzed by the Evaluation Unit. n = 348

Figure 16: Position level by funding status (Vanier CGS recipients and applicants) – Non-academic sectors

Figure 16 – Long description
  Executive-level Managerial-level Senior-level Intermediate-level Junior-level Other, please specify Don't know/
prefer not to answer		 Total
CIHR Applicants 0% 16% 18% 40% 15% 5% 5% 34
CIHR Recipients 12% 7% 9% 33% 23% 14% 2% 43
NSERC Applicants 0% 20% 24% 32% 16% 3% 5% 35
NSERC Recipients 8% 19% 12% 38% 19% 4% 0% 26
SSHRC Applicants 17% 9% 28% 17% 4% 9% 15% 22
SSHRC Recipients 9% 9% 22% 26% 9% 13% 13% 23
Male Applicants 0% 18% 25% 36% 10% 2% 8% 43
Male Recipients 11% 9% 6% 34% 26% 6% 9% 47
Female Applicants 8% 15% 20% 27% 16% 8% 7% 48
Female Recipients 9% 13% 20% 31% 11% 16% 0% 45
Applicants (n = 91) 4% 16% 23% 31% 13% 5% 8% 91
Recipients (n = 92) 10% 11% 13% 33% 18% 11% 4% 92

Source: Survey of Vanier CGS recipients and applicants, 2019.

Appendix B – Methodology

Additional details about the multiple lines of evidence used in the evaluation are presented in this section. They included a document and data review, analysis of administrative data, VEAR data, a recipient and applicant tracer survey, and in-depth interviews with KIs.

Document Review

A document and data review was conducted to provide context to the evaluation and help address several evaluation questions relating to program relevance and performance. Documentation reviewed included Vanier CGS program literature, minutes of meetings of governance bodies including the TAP-MC and TAP-SC and previous evaluation reports (e.g., the Vanier CGS program evaluation, 2014, the Banting Postdoctoral Fellowship program evaluation, 2015, and agency-specific doctoral program evaluations).

Administrative Data Analysis

A review of Vanier CGS program records and administrative data from the VBS provided information on institution scholarship allocations and proportions fulfilled, and competition data such as nominee success rates and their distribution by sex, and Canadian citizenship status, review scores and program expenditures which helped contextualize the program. This analysis also informed the sampling strategies used in the key informant interviews and the recipient and applicant tracer survey.

Vanier End of Award Report (VEAR)

Vanier award recipients are expected to complete the VEAR within 18 months of award expiry. The VEAR assesses recipients’ outputs and outcomes during the tenure of the award in terms of the development of leadership potential, academic outputs and satisfaction with program design elements.

Data from 348 Vanier CGS award recipients was analyzed for the purposes of this evaluation. The sample covered seven cohorts (2010-11 to 2016-17) and consisted of 110 CIHR-funded recipients (32%), 114 NSERC-funded recipients (33%) and 111 SSHRC-funded recipients (32%). The remaining 13 recipients did not specify their funding agency. There were 249 Canadian citizens and permanent residents (70%) and 103 citizens and permanent residents of other countries (30%). According to administrative data on sex, females (48%) slightly outnumbered males (42%). In terms of first official language, the majority were English (76%), compared to French (22%). VEAR participation and corresponding response rates are presented in Figure 13: Recipient's collaboration with researchers and trainees during the Vanier CGS.

Analysis of the VEAR data included overall frequencies for each question, as well as disaggregated by other relevant population variables, including funding status (recipient and applicant), agency (CIHR, NSERC, and SSHRC), sex (male and female), and citizenship status (citizen or permanent resident of Canada vs. citizen or permanent resident of foreign country). It should be noted that the results from the VEAR are based on self-report data and there was no comparison group for these outcomes.

Recipient and Applicant Tracer Survey

An online survey targeting the first four cohorts of Vanier CGS applicants and recipients (2009-10 to 2011-12) was conducted by Goss Gilroy Inc. on behalf of the Evaluation Team.

Data from 508 Vanier CGS award recipients and applicants was analyzed for the purposes of this evaluation. The sample covered four cohorts (2008-09 to 2011-12) and included both recipients and applicants from each of the three funding agencies, including 181 from CIHR (36%), 149 from NSERC (29%) and 178 from SSHRC (35%). There were 368 Canadian citizens (72%), 36 Canadian permanent residents (7%) and 103 citizens and permanent residents of other countries (20%). According to administrative data on sex, females (60%) outnumbered males (40%). In terms of first official language, the majority of Canadian citizens and permanent residents were English (74%), compared to French (26%). Less than one-quarter of the sample identify as a member of a visible minority (19%), while approximately three-quarters of the sample do not (79%). Only 1% of the Vanier applicants and recipients in the sample identify as Indigenous (First Nation, Métis or Inuk), while 98% do not. Very few Vanier applicants and recipients in the sample report having one or more disabilities (7%), while the majority have none (91%). Tracer survey participation and corresponding response rates are presented in Figure 17a: Tracer survey response rates.

The tracer survey data were weighted for application status and citizenship status (applicants only). Analysis of the survey data included overall frequencies for each question, as well as disaggregated by other relevant population variables, including funding status (recipient and applicant), agency (CIHR, NSERC, and SSHRC), sex and citizenship status.

Figure 17a: Tracer survey response rates
Total Recipients Applicants
Total sample 1757 603 1154
Total valid sample (Total sample - Bounce backs) 1534 535 999
Number of responses received 508 261 247
Response rate 33.1% 48.8% 24.7%

Source: Goss Gilroy Inc. Vanier CGS Evaluation - Survey Field Report, 2019.

The sample disposition is presented by competition year, agency, sex, citizenship status and language preference in Figure 17b: Tracer survey sample disposition.

Figure 17b: Tracer survey sample disposition
Overall Recipients Applicants
Status   Number (%) 508 (100%) 261 (51%) 247 (49%)
Competition Year Total Number (%) 508 (100%) 261 (100%) 247 (100%)
2008-2009 Number (%) 102 (20%) 66 (25%) 36 (15%)
2009-2010 Number (%) 114 (22%) 73 (28%) 41 (17%)
2010-2011 Number (%) 99 (20%) 47 (18%) 52 (21%)
2011-2012 Number (%) 193 (38%) 75 (29%) 118 (48%)
Agency Total Number (%) 508 (100%) 261 (100%) 247 (100%)
CIHR Number (%) 181 (36%) 103 (40%) 78 (32%)
NSERC Number (%) 149 (29%) 66 (25%) 83 (34%)
SSHRC Number (%) 178 (35%) 92 (35%) 86 (35%)
Sex Total Number (%) 508 (100%) 261 (100%) 247 (100%)
Male Number (%) 203 (40%) 106 (41%) 97 (39%)
Female Number (%) 305 (60%) 155 (59%) 150 (61%)
Citizenship Status Total Number (%) 508 (100%) 261 (100%) 247 (100%)
Canadian citizen Number (%) 368 (72%) 189 (72%) 179 (73%)
Permanent resident of Canada Number (%) 36 (7%) 17 (7%) 19 (8%)
Citizen/Permanent resident of another country Number (%) 103 (20%) 55 (21%) 48 (19%)
No status Number (%) 1 (0%) 0 (0%) 1 (0%)
Language preference Total Number (%) 508 (100%) 261 (100%) 247 (100%)
English Number (%) 374 (74%) 192 (74%) 182 (74%)
French Number (%) 134 (26%) 69 (26%) 65 (26%)

Source: Adapted from Goss Gilroy Inc. Vanier CGS Evaluation - Survey Field Report, 2019.

Key Informant Interviews

Qualitative data obtained from key informant interviews was used to help address several evaluation questions and provide insight into the review process at the institution level.

A total of 48 interviews were conducted. Thirty-one interviews were conducted by Ference & Company with award recipients and applicants, TAP-SC members and Vanier CGS program management. The remaining 17 interviews were conducted by the CIHR Evaluation Unit with academic supervisors, scholarship liaison officers, selection committee members and chairs, and a Global Affairs Canada representative. Interview participation and response rates by target group are outlined in Figure 18: Key informant interview participation by respondent group.

Figure 18: Key informant interview participation by respondent group
Interview Target Group Contacted (N) Target (N) Interviewed (n) Response Rate (%)
Vanier CGS recipients 140 12 14 10%
Vanier CGS applicants 178 12 12 7%
Tri-agency Programs – Steering Committee (TAP-SC) 6 5 4 67%
Vanier CGS program management 1 1 1 100%
Subtotal: Ference & Company Interviews 326 28 31 110%
Academic supervisors 17 6 5 83%
Scholarship liaison officers 6 5 4 80%
Selection committee chairs 5 3 3 100%
Selection committee members 18 6 4 67%
Global Affairs Canada representative 1 1 1 100%
Subtotal: CIHR Evaluation Unit Interviews 47 21 17 81%
Total 373 49 47 96%

Source: Ference & Company, Vanier CGS Evaluation - Key Informant Interviews Technical Report

Reporting Approach

For the purposes of reporting key findings of the analysis of VEAR data, tracer survey data and key informant interview qualitative findings, the following response summary table (Figure 19: Data Response Summary) was developed and applied based on the proportion of respondents who provided a particular answer or identified a theme.

Figure 19: Data Response Summary
Response Summary % of Cases
All 100%
Almost all 90%-99%
Majority 80%-89%
Approximately three-quarters 70%-79%
Approximately two-thirds 60%-69%
Slightly over half 55%-59%
Approximately half 46%-54%
Slightly less than half 40%-45%
Approximately one-third 30%-39%
Approximately one-quarter 20%-29%
Some 10%-19%
A few or small number 5%-9%
Date modified: