Evaluation of the Vanier Canada Graduate Scholarships Program

Final Report 2014

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor Address Locator 4809A
Ottawa, Ontario K1A 0W9 Canada

Acknowledgements

This report was authored by Patrick MacGuire. The evaluation was carried out by an Evaluation Working Group which, during the different phases of the evaluation, included: Patrick MacGuire, Michael Goodyer, Shannon Clark-Larkin, Nicole Michaud, Joanne Tucker, Heather Bell, Kwadwo (Nana) Bosompra, Christopher Manuel, Igor Tupitsyn, Marc-Étienne Joseph and Rachele Goulet.

Special thanks to all the participants in this evaluation and to Circum Network Inc., Nicole Michaud and Joanne Tucker for assisting with data collection and analysis.

Table of Contents

Executive Summary

The Vanier CGS program was created by the government of Canada in 2008 to strengthen Canada’s ability to attract and retain the world’s top doctoral students and establish Canada as a global centre of excellence in research and higher learning. The program is implemented through the three Federal Granting agencies and invests approximately $25 million annually to support 500 Canadian and international doctoral students studying at Canadian universities.

This is the first evaluation of the Vanier CGS program and was led by the Canadian Institutes of Health Research (CIHR) in partnership with the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC). The evaluation examines the relevance and performance of the program as well as its design and delivery in order to meet Treasury Board requirements for evaluation and inform the renewal of its Terms and Conditions which expire July 31, 2014.

Key Findings

Relevance

Alignment with Government Roles, Responsibilities and Priorities
Continued Need

Performance

Attraction and Recruitment
World-Class Students
Impact of Vanier CGS on Scholars
Training
Retention and Career Outcomes
Vanier Alumni Network
Impact of Vanier CGS on Canadian Universities
Prestige
Design and Delivery
Vanier CGS Cost-Efficiency

Conclusions

Evaluation findings indicate that the program is supporting world-class doctoral students – the majority of whom are Canadian, with most Vanier scholars demonstrating exceptional leadership in the area of research. The Vanier CGS award is having the strongest positive impact on students’ financial situation, their need for obtaining income during their studies and the time they were able to devote to their education.

The evaluation found that the Vanier CGS program is considered to be a highly prestigious award in Canada and the presence of Vanier scholars enhances the reputation of a faculty, program or laboratory but has little to no impact on the reputation of the institution. The majority of Vanier scholars are satisfied with their training and skill development and most report that their training has been useful in preparing them for their career. Of the Vanier scholars who have completed their studies, the majority are employed in the university sector and are living in Canada.

Findings from the evaluation regarding the communication preferences of Vanier scholars can help inform the development of an alumni network of Vanier scholars, which is an expected output of the program that has not yet been implemented.

The evaluation identified key barriers that negatively impact its ability to attract and recruit students from outside of Canada: the inability of institutions to guarantee receipt of the award, the high competitiveness of the program in relation to the relatively small number of scholarships awarded and the reluctance of doctoral supervisors to nominate students with whom they have not yet worked with and who are not already enrolled in the institution in which they are seeking Vanier support.

Opinions on the design and delivery of the program were positive overall, with the majority of Vanier scholars reporting satisfaction with the award amount and duration, however the evaluation found that the three year nomination allocation system is contributing to the reluctance of institutions to nominate new students (including foreign candidates) which limits the ability of the program to attract students from outside the country and has had a negative impact on the quality of candidates put forward for nomination. As a result, many universities suggested that the three year allocation cycle be changed to annual nominations.

As the objective of the Vanier CGS program is to be an internationally competitive scholarship program that enables Canada to attract world-class doctoral students (a key feature which distinguishes it from other federal scholarship programs) and given the reluctance of institutions to nominate foreign candidates, it is important that a portion of allocations be targeted to international students outside of the country in order to facilitate their nomination and better enable the program to meet its objective in terms of attraction. As well, the program’s competition deadlines were viewed by many as being problematic in terms of attracting new (including foreign) students to Canadian institutions and, as such, should be reconsidered.

The interpretation and assessment of the leadership criterion was identified as a challenge and, in light of the program’s expected long-term outcome that Vanier scholars become leaders in Canada and abroad, there is a need to further develop and clarify the definition and assessment of leadership given the important role it plays in the selection process at the university and federal granting agencies.

While the award amount is meeting the educational-related financial needs of almost all recipients, questions remain as to the incremental outcomes that result from the additional financial value of the Vanier award in comparison to other doctoral scholarship awards given the absence of comparator data at the time of this study. There is a need therefore to pursue an additional analysis using comparator data obtained through the evaluations of other doctoral award programs to identify the incremental outcomes that result from the additional financial support offered through the Vanier scholarship.

Recommendations

  1. To enable the Vanier CGS program to better meet its objective of attracting and recruiting world-class doctoral students to Canadian universities, the following changes to the allocation and application processes should be considered:
    1. The three year allocation cycle should be changed to annual allocations with restrictions placed on carrying forward unused quotas.
    2. A portion of nomination allocations should be targeted for foreign students not already enrolled in the institution in which they are seeking Vanier support, with the amount calculated, at least in part, on an institution’s international student enrollment rate. The Vanier CGS program should monitor the extent to which the target of foreign student nominees is being met, per institution and in total, after each competition and use this information as part of the calculation to determine targets and allocations for the next competition year.
    3. The timing of the application deadlines for the Vanier CGS program should be re-considered in light of findings from the evaluation.
  2. To help improve the assessment of the leadership criterion in the University and Federal Granting Agency selection processes and increase the extent to which it is interpreted consistently, Vanier CGS program management should establish a clearer definition of what leadership is and how it should be evaluated – especially for foreign students. Suggestions for improvement obtained through the evaluation should be taken into consideration and any changes made should be communicated to universities and Federal Granting Agency selection committee members.
  3. To help inform the implementation of a Vanier Alumni Network, data obtained through the evaluation on the communication preferences of Vanier scholars for interaction with other award holders and program staff should be considered to ensure that Vanier CGS graduates are connected to the program and to other scholars after completion of their studies.
  4. Evaluation findings demonstrate that the Vanier program is meeting the educational-related financial needs of almost all of its recipients and that it is viewed as a highly prestigious award in Canada. However, the evaluation was unable to assess what incremental outcomes are associated with the higher value of the scholarship in comparison to the CGS and Federal Granting Agency doctoral awards as evaluations of those programs were in progress at the time of this study and comparison data was not yet available. The surveys used in these scholarship evaluations were designed to enable a comparative analysis of data across programs. As such, it will be important to undertake a further analysis to assess the similarities and differences in the results achieved across programs as it can provide insight into what effect features of the Vanier scholarship, most importantly its award amount, have on outcomes. Issues to be examined would include incentives to enroll in studies, financial situation of students, training (including skill acquisition), research productivity, retention rates and employment. Based on this, it is recommended that Vanier CGS program management consider the results of this analysis in the future planning and design of the program.

Background

Photo: 2011 Vanier CGS Awards Announcement - McMaster University. Photo courtesy of the Vanier Banting Secretariat.

Vanier CGS Program

Introduced in Budget 2008 (Department of Finance Canada, 2008, p. 114) with the first round of scholarships awarded in 2009, the Vanier CGS program was created to strengthen Canada’s ability to attract and retain the world’s top doctoral students and establish Canada as a global centre of excellence in research and higher learning. Vanier scholarships are branded and promoted as a key instrument to achieve Canada’s Science and Technology Strategy (Industry Canada, 2007) goal of increasing the supply of highly qualified and globally connected graduates and build a People Advantage for Canada.

Vanier invests approximately $25 million annually to support 500 Canadian and international doctoral students studying at Canadian universities. As of 2014, approximately 1,000 scholarships valued at $50,000 per year for three years have been awarded through the program.Footnote 1 The Vanier program is administered through the three federal granting agencies: the Canadian Institutes of Health Research (CIHR), the Social Sciences and Humanities Research Council (SSHRC) and the Natural Sciences and Engineering Research Council (NSERC).

Figure 1.1 – Vanier CGS Award Details and Program Objectives

Award Details

Program Objectives

Source: Vanier CGS Terms and Conditions; Vanier CGS Performance Measurement Strategy

The following are the key stakeholders for the Vanier CGS program:

Vanier CGS Application and Selection Process

Vanier scholarships are awarded after a competitive selection process involving reviews conducted at eligible Canadian universities as well as by the federal research granting agencies.Footnote 2

University Review

Students must be nominated for a Vanier CGS award by a recognized Canadian university. Nominations are initiated in one of two ways: (1) the candidate informs the faculty of graduate studies at the selected university of their intent to apply or (2) the university initiates the nomination process by contacting the candidate. The scholarship liaison officer at each eligible Canadian university is responsible for coordinating the university review of their selected candidates and forwarding nominations for Vanier CGS awards to the appropriate federal research granting agencies in accordance with their university's allocationFootnote 3. In total, 1,800 nominations are allocated over three years to eligible institutions (200 per granting agency per year).Footnote 4

Federal Granting Agency Selection Committees

Once submitted to the appropriate granting agency, each nomination is evaluated by an agency-specific selection committee. Per year, each selection committee recommends the top 55 or 56 candidates (for a total of approximately 167 candidates among the three granting agencies) to the program’s Steering Committee based on the nominee's academic and research potential, as well as their leadership skills.

Steering Committee

The Steering Committee, which comprises the presidents of the three federal granting agencies and the deputy ministers of Industry Canada and Health Canada, makes the final decisions on funding. Each year, the Steering Committee approves approximately 167 Vanier scholars for funding based on the recommendations of the selection committees.

The Vanier Banting Secretariat

The Vanier Banting Secretariat was implemented in October 2012 and is responsible for providing administrative oversight to the Vanier CGS and Banting Postdoctoral Fellowships programs. The secretariat is located at CIHR and consists of an executive director, manager and program delivery staff from each of the federal granting agencies.

Figure 1.2 – Vanier CGS Logic Model

Figure 1.2: long description

Evaluation Purpose and Scope

This study is the first evaluation of the Vanier CGS program and was led by the Canadian Institutes of Health Research (CIHR) in partnership with the Natural Sciences and Engineering Research Council (NSERC) and the Social Science and Humanities Research Council (SSHRC). The evaluation examines the relevance and performance of the program as well as its design and delivery in order to meet Treasury Board requirements for evaluation (see Appendix for a description of TBS requirements and for a cross-walk of evaluation questions with TBS requirements) and inform the renewal of its Terms and Conditions which expire July 31, 2014. The evaluation covers the time period from the program’s inception in 2008–2009 to the end of fiscal year 2012–2013.

As two immediate outcomes are expected to occur within four years after inception of the program (world-class students are attracted and recruited and world-class students are trained), the evaluation will assess the extent to which these two outcomes have been achieved as well as examine the progress made towards the achievement of longer-term outcomes.

Key Findings

Relevance

Need for Vanier CGS

Evaluation Question

6. Is there a continued need for the Vanier CGS program?

One indicator of continued need for the Vanier CGS program is determining whether its demand (in terms of number of applicants applying for nomination) has increased, decreased or stayed the same over time. Data on the total number of applicants indicates the demand for the program has decreased since the program’s harmonized launch through the federal granting agencies in 2009 (Figure 2.1) although the variance across most years is relatively minor.

Figure 2.1 – Number of Vanier CGS Applicants by Year

Source: ResearchNet

Figure 2.1: long description

Another indicator of demand is the proportion of Vanier allocation quotas fulfilled over the period of the first allocation cycle (2010–2012). Under this period, a total of 52 Canadian universities received at least one nomination allocation. At the end of the cycle, only 21% of institutions fulfilled their allocation quota, which implies that most institutions were unable to identify enough high calibre candidates for nomination against the maximum amount set out by the federal granting agencies over a three year period. Furthermore, of the institutions who did not fulfill their quota, 8% did not submit a single nomination. Finally, of a total of 4,106 applicants since 2009, 2,120 or a little over half were screened out of the university nomination selection process (Figure 2.2). These results are concerning given that the Vanier CGS award is open to students worldwide yet 8 out of 10 institutions are not able to identify enough suitable candidates to fulfill their nomination quotas. As institutions screened out a little over half of all applicants in the first stage of the selection process, it can be concluded that unfilled quotas were not due to a lack of interested candidates to consider for nomination.

Figure 2.2 – Vanier CGS Application Data 2008–2012

Source: ResearchNet

Figure 2.2: long description

*For the first competition of the Vanier CGS program in 2008, the application and peer review process was implemented differently across federal granting agencies. As such, data on unsuccessful applicants in 2008 is not equivalent and comparable to other Vanier CGS competition years and has been excluded from the analysis presented in Figure 2.2 as a result.

A final indicator of program demand examined through this study is the proportion of those offered the Vanier CGS award that declined. Data obtained from administrative records shows that, over the period of 2008–2012, only 22 (3%) successful candidates declined the Vanier award (Table 2.1). Given that Vanier recipients are ineligible to hold other federal granting agency scholarships concurrently, the small number of declines implies that Vanier CGS is the top choice for financial support for doctoral students amongst the slate of doctoral awards offered through CIHR, NSERC and SSHRC. This finding is not surprising given the relatively high dollar value of the Vanier scholarship.

Table 2.1 – Vanier CGS Declines 2008–2012
2008 2009 2010 2011 2012 Total
CIHR 1 3 0 2 1 7
NSERC 1 2 2 1 0 6
SSHRC 3 1 2 2 1 9
Vanier CGS Total 5 6 4 5 2 22

Source: Federal Granting Agency Administrative Data

In terms of addressing the education-related financial need of doctoral students, evaluation findings indicate that the Vanier scholarship is sufficient for the majority of its recipients however some stakeholders feel that the award amount may be excessive. A survey of Vanier scholars revealed that only 5% were not concerned about their financial status at the time of their application to their doctoral program and close to half (43%) expressed a high degree of concern which implies the existence of a demonstrated need for financial support for Vanier scholars at the outset of their studies.

As to whether the education-related financial need of Vanier scholars is being addressed, of those who have not yet completed their studies, 8 out of 10 (82%) reported having no debt related to their doctoral program. For those who completed their doctoral studies, 9 out of 10 (90%) reported having no debt upon graduation. This finding holds true even when comparing Canadian and foreignFootnote 5 students (who can incur higher costs to study in Canada). As noted elsewhere in the report, the strongest positive impact that Vanier scholars felt the award had was related to the value of the award and its impact on their current financial situation (for those who have not yet graduated) as well as their need for obtaining income during their studies.

Although evaluation findings indicate that the Vanier award is having a positive impact financially on its recipients and, as noted elsewhere in the report, is viewed as being highly prestigious, other findings obtained through the evaluation raise uncertainty as to whether the value of the award is commensurate to the financial needs of its recipients. Specifically, the proportion of Vanier scholars who were debt free are in line with the overall proportion of Canadian doctoral graduates (88%) who have reported not owing any money related to their graduate studies (Statistics Canada, 2008). Although proportions of students debt-free are similar, it may be that the Vanier scholarship offsets the extent to which employment and financial support is sought by students as compared to Canadian doctoral students in general as the award was reported to have a very positive impact on recipients’ need for obtaining income during their studies.

Photo: 2009 Inaugural Vanier CGS Awards Announcement – Ottawa, Ontario. Photo courtesy of the Vanier Banting Secretariat.

As well, survey findings show that almost all award recipients (92%) expressed the highest level of satisfaction for the dollar amount of the Vanier award (with proportions consistent across fields of study). Finally, when asked for suggestions for improvement to the program, some universities representatives and Vanier supervisors as well as a small percentage (5%) of award recipients felt that the Vanier award amount was too high. Some argued that a portion of the funding should be allowed to cover research and travel-related costs (e.g., to attend conferences) while others suggested that having more CGS awards would be preferable. As comparator data on CGS and other Federal Granting Agency doctoral scholarship programs was unavailable at time of study (see methodology section for further details), it is important that further analysis be conducted to assess the incremental outcomes that are associated with the higher financial value of the Vanier award.

Evaluation findings indicate that a majority of Vanier recipient graduates are working in the university sector – a trend similar to the early stage career trajectory of Canadian doctoral graduates in general. According to a report from Statistics Canada based on data from the 2005 Survey of Earned Doctorates and the 2007 National Graduates Survey (Statistics Canada, 2011), a little over half (56%) of Canadian doctoral graduates living in Canada and the United States two years after graduation in 2005 were employed in educational services with the vast majority of them working in a university (87%). Data obtained through this evaluation shows that the majority of Vanier recipients who have not yet completed their degree intend to work in the university sector (79%) with 84% expressing interest in pursuing a position as a university professor/researcher. For Vanier scholars who have completed their studies, 74% indicated that they are employed in the university sector (with many working in postdoctoral positions).

As to whether there is sufficient demand in future years in Canada for the type of employment the majority of Vanier scholars intend to pursue after graduation, Employment and Social Development Canada (2011) predicts that supply and demand for university professor and assistant positions will be balanced over the coming years with shortages in some fields which is promising in terms of retaining Vanier graduates seeking employment in these positions. Specifically, the number of job openings is expected to be smaller over the period of 2011–2020 as compared to 2001–2010 with the number of job seekers expected to increase slightly. As well, according to projections from ESDC, most PhD graduates may eventually become university professors or assistants, which could cause a rapid rise in the number of job seekers. However, ESDC notes that based on projections and considering that labor supply and demand in this occupation were balanced, it is expected that supply and demand will continue to be balanced over the 2011–2020 period and there may be shortages in certain fields of study.

Given the career aspirations of Vanier scholars, the need for the program will be determined at least in part on the demand for professor positions in Canada. If the supply exceeds demand, there is a risk that the program will fund a large majority of students who eventually leave the country to seek employment elsewhere.

Evaluation Question

7. Does the Vanier CGS program align with government priorities and is it consistent with federal roles and responsibilities?

Alignment with Federal Government Priorities

Introduced in the Federal Government’s 2008 budget (Department of Finance Canada, 2008, p. 114), the Vanier CGS program was an initiative designed to attract the world’s best doctoral students to Canadian institutions. The program has remained a federal government priority since its inception and the most recent 2014 federal budget reiterated its role in Canada’s science, technology and innovation system and referenced the Vanier CGS program as one of several means of government support for post-secondary education (Department of Finance Canada, 2014, p. 75).

When it comes to the importance of the Vanier scholarship in addressing federal government priorities, some granting agency representatives interviewed through the evaluation thought that it depended on the nature of the goal that the program was to serve. According to them, if the goal was to create leaders and researchers, then there is a need for a separate leadership building program that is currently not being met (because research excellence and leadership do not necessarily come hand in hand).

Interviewees also thought that the Vanier program was part of a greater Government of Canada effort to brand itself as valuing research. Some argued that, while the Vanier program played an important role in meeting this goal, it was not an optimal tool to develop research capacity, due to the limited number of awards handed out per year. The role and value of the program therefore need to be assessed against a specific goal.

Compatibility with Federal Granting Agency Priorities

The goals of the Vanier CGS program align seamlessly with the research capacity building priorities of the three Federal Granting Agencies – CIHR, NSERC and SSHRC. CIHR aims to ‘invest in world-class research excellence through attracting and retaining the best international scholars and experts’ (CIHR, 2009). NSERC aims to attract, retain and develop highly qualified people in natural sciences and engineering (NSERC, 2014) while SSHRC strives to ‘make Canada a world leader in social sciences and humanities research and research training’ (SSHRC, 2013).

Alignment with Federal Roles and Responsibilities

The importance of the spill-over effects of graduate studies as a reason for the Government of Canada to support world-class doctoral students was underscored by key informant interviewees. In that sense, the impacts of graduate training were perceived as considerable and social impact comes from better educated students rather than patents and licenses. Therefore, they felt that the federal government’s involvement in supporting world-class doctoral students is a sign of its true ownership of the talent advantage priority.

Performance

Achievement of Immediate Outcomes - Capacity Development

Evaluation Question

1. To what extent has the Vanier CGS program achieved its expected immediate outcomes?

1.1. Has the Vanier CGS program attracted and recruited world-class students?

Attraction and Recruitment

Evaluation findings present strong evidence that the Vanier CGS program has not fully met its objective of attracting and recruiting world-class students. To date, the majority of awards have been given to students already enrolled in Canadian institutions at the time of application to the program. The ability of the program to attract and recruit students from outside of Canada has been limited due to the fact that potential international applicants face barriers in terms of obtaining nomination support from Vanier eligible institutions. As a result, the program has not met its unique objective (among federal granting agency scholarship awards) of attracting and recruiting the world’s best doctoral students to Canada.

Survey results reveal that the majority of Vanier scholars (86%) reported living in Canada at the time of their application to the award and that 13% of Vanier scholars were foreign students (Table 2.2). These results are concerning given that the proportion of foreign students receiving the award annually is lower than the international student enrollment rate of 22% for doctoral and postdoctoral programs in Canada in 2010 and the fact that international student enrollment has outpaced that of Canadian students over the period 1998 to 2010 (Statistics Canada, 2014). Of the number of foreign Vanier scholars, close to one-quarter (24%) were living in the United States when they applied. In total, foreign Vanier scholars were attracted from 40 different countries worldwide.

Figure 2.3 – % of Foreign Vanier Scholars by Year

Source: Survey of Vanier Scholars

Figure 2.3: long description

Table 2.2 – % of Foreign Vanier Scholars by Year
2008 2009 2010 2011 2012 Total
Health Sciences 3% 8% 3% 11% 10% 7%
Natural Sciences & Engineering 19% 8% 14% 11% 16% 13%
Social Sciences & Humanities 12% 15% 28% 10% 19% 17%
Vanier CGS Total 11% 10% 16% 11% 15% 13%

Source: Survey of Vanier Scholars

Another important finding from the evaluation is that, at the time of application to the award, a total of 93% of Vanier scholars were already studying at the institution in which they were seeking Vanier support (approximately 70% of Vanier scholars were already enrolled in their doctoral program) (Table 2.3).

Table 2.3 – % of Vanier Scholars Already Enrolled in Canadian Doctoral Program at Time of Scholarship Application
2009* 2010 2011 2012 Total
CIHR 66% 68% 63% 82% 70%
NSERC 58% 63% 64% 70% 64%
SSHRC 67% 78% 82% 82% 77%
Vanier CGS Total 64% 69% 69% 78% 70%

Source: ResearchNet (N=669)

* data pertaining to the institution for the degree in progress of Vanier applicants was not available for 2008.

** applicants were determined to be already enrolled at the Canadian institution for which they were seeking Vanier CGS support at the time of application to the award if their degree in progress was a doctoral/PhD program and the institutions for degree in progress and degree sought were the same.

< td class="text-center" headers="header30 header35 header42">116
Table 2.4 – Demographic Characteristics of Vanier Scholars 2008–2012
Gender
(N=811)		 Mean Age
(N=756)		 Language
(N=830)		 Citizenship*
(N=830)
Female Male English French C R O
CIHR 160 27 ± 4 266 14 216 12 52
NSERC 103 167 26 ± 3 257 20 180 11 86
SSHRC 179 86 30 ± 6 214 59 167 12 94
Vanier CGS Total 442 369 28 ± 5 737 93 563 35 232

Source: ResearchNet

* C=Canadian Citizen, R=Permanent Resident of Canada, O=Other Citizenship(s).

As part of the evaluation, key stakeholders of the Vanier program were interviewed including representatives from the Department of Foreign Affairs, Trade and Development Canada (DFATD), formerly Foreign Affairs and International Trade Canada (DFAIT), as well as deans of graduate studies and research liaison officers from 23 of the 52 Canadian universities under the first allocation cycle. Findings from this line of evidence helped provide insight into why the Vanier CGS program is mostly supporting Canadian doctoral students already enrolled in their studies.

Interview respondents doubted Vanier's ability to attract new international students (who are not already living in Canada), because of its high level of competitiveness and long competition timelines. DFATD interviewees mentioned that they were unable to attract new students with a Vanier award because they could make no guarantees that potential candidates would be successful in receiving it. University respondents also confirmed that they are unable to use the award to recruit as the number given out is too small and the process is extremely competitive which prevents them from making any guarantees to doctoral students (prospective or already enrolled) about receiving the award. As well, some university representatives believe that international students who are nominated tend to keep their options open, so it is less likely that they would come to Canada for their doctoral studies without a Vanier scholarship.

The role of the Vanier award as a recruitment tool for students (both domestic and international) is furthermore limited by doctoral supervisors’ reluctance to nominate students with whom they have not yet worked with, as would be the case if students were to be nominated during their first year of doctoral studies. The program timelines are such that between the application period and when the decision is made at the federal granting agency level, first year candidates would have had to apply to Vanier a full year before starting their doctoral studies. As such, supervisors have had very limited contact with the most recent cohort of students by the time the applications need to be submitted, and they are less likely to put forward students who are starting their doctoral studies than those who are in their second year, or who have worked with this supervisor during their Masters' (and who are therefore not new to the institution), or who have been students of close colleagues in other centres. Results from a survey of doctoral supervisors of Vanier scholars confirms this finding as only 9% reported having low to no familiarity with their award recipient at the time of their application to the scholarship.

Vanier CGS selection committee members who participated in the evaluation through an expert panel review agreed that international applications have been flagged as a longstanding concern by the granting agency selection committees because foreign students have more hurdles to overcome in terms of securing a nomination for a Vanier CGS award, given the need to establish support and connection to a Canadian institution. Concern was also expressed by selection committee members that the method of nomination at Canadian universities has been overly restrictive and detrimental to attracting world-class applicants, resulting in the narrowing of the potential pool. In addition, concern was raised by the panel that current timelines for application submission and award decision might be potentially limiting the applicant pool.

As an incentive to enroll in doctoral studies in Canada, evaluation findings show that the existence of the Vanier CGS program appears to have played a moderate to high degree of importance in the decision of Vanier scholars - Canadian and foreign - for about half of this group (53% for Canadians and 51% for foreigns). In addition, only a small number of Vanier scholars (7%) reported that they would have pursued doctoral studies outside of Canada if they had not received the scholarship. This finding is most likely due in large part to the fact that 70% of Vanier scholars (excluding those funded in 2008) were already enrolled in doctoral studies at the time they applied for the award (Table 2.3).

In making their decision to pursue doctoral studies in Canada, the reputation of their primary supervisor was identified by approximately 8 out of 10 Vanier scholars as having a moderate to high influence on their decision (this finding holds true for both foreign and Canadian students). The reputation of their university was also identified by the same proportion of award holders as having been a factor in their decision although fewer rated it as having high importance. Other important factors identified by the majority of Vanier scholars included the reputation of their department, the university being in Canada and the quality of life in Canada.

World-Class Students

To help address the issue of whether the program has been attracting and recruiting world-class students, the evaluation obtained the perspectives of current and former Vanier selection committee members through an expert panel review (divided into three sub-committees each representing health, natural sciences and engineering and social sciences and humanities) that examined a purposive sample of Vanier CGS nominees and recipients’ applications over the period 2008–2012, with a view to assessing their overall quality in terms of being world-class. The concept of world-class in relation to the Vanier program was deemed as a synonym for excellence by expert panel members.

Photo: 2009 Inaugural Vanier CGS Awards Announcement – Ottawa, Ontario. Photo courtesy of the Vanier Banting Secretariat.

In reviewing a total of seventy-five files per sub-committee, the panel concluded that the overall quality of both successful and unsuccessful applications was consistent over time and that their scores given as part of the panel reflected the judgments made by the selection committees (although a few cases were noted by members of the panel where their scoring differed from that of the selection committee in terms of applications that were funded but would not have been and those that were not but would have been). These findings should be interpreted within context, however, because the assessment of the overall results of the expert review panel was based on a small sample of Vanier CGS applications (with smaller subsets for each panel member). In responding to the issue of the overall quality of applicants, panel members expressed some reservation as to whether all Vanier CGS recipients were indeed world-class (although they noted that they lacked data on what candidates went on to do after their application to the award and thus were limited in terms of the evidence needed in making a sound decision on quality).

Further evidence on the quality of doctoral students awarded a Vanier scholarship was obtained through a survey of their supervisors where respondents were asked to rate their Vanier scholars on various research, teaching and service-related dimensions. In terms of leadership, findings showed that Vanier scholar supervisors rated the award holder they supervised/were supervisingFootnote 6 as being exceptionalFootnote 7 at demonstrating leadership in research (62%), personal and/or professional leadership (51%), leadership in providing service to the academic community (31%), leadership in teaching (27%) and leadership in providing service to the larger community (21%). In addition, results show that a higher percentage of supervisors rated their Vanier scholars as demonstrating exceptionalism on several research, teaching and service-related dimensions compared to other doctoral students whom they supervised. Of particular importance, 31% indicated that Vanier scholars demonstrated an exceptional ability to lead/influence others. These findings imply that the majority of Vanier award holders, while being rated higher than other doctoral students, are mainly exhibiting leadership in research; however according to supervisors, their leadership potential in other areas has not yet been fulfilled for the majority of recipients (Table 2.5).

Table 2.5 – Supervisor Ratings of Vanier Scholars Compared to Other Doctoral Students Supervised
% of Supervisors Providing Ratings of Exceptional For:
Dimensions Vanier Scholar Supervised Other Doctoral Students Supervised (N=239)
Quality of research 58% 13%
Productivity in the research environment 52% 10%
Publication and knowledge dissemination record 46% 8%
Ability to create a community of peers 32% 9%
Ability to create collaborative relationships 40% 8%
Ability to supervise and mentor 25% 10%
Ability to manage their projects and research 51% 11%
Ability to teach 25% 5%
Ability to lead/influence others 31% 6%
Contributions to the academic community 24% 5%
Contributions to the larger community 22% 5%

Source: Survey of Vanier Scholar Supervisors

University representatives interviewed were also asked to assess the extent to which those nominated and awarded were world-class. It was generally agreed the Vanier award had set expectations of excellence, and that therefore students who applied and were nominated were usually world-class. On the other hand, some interviewees did note that, while students who were awarded a Vanier were world-class, nominees, albeit all high-performance students, were not all world-class. They noted that the leadership criterion appears to be a key distinguishing factor: nominees may well all be excellent researchers but awardees distinguish themselves by also being or having a strong potential to be leaders in their environment. Said one university representative: "Of our general pool of students, world-class students are pretty rare, which is statistically logical. For those selected for the Vanier, we never had issues finding students who were at least very strong academically. The real issue came with meeting all the other criteria, mainly the leadership skills."

Awareness of the Vanier CGS Program

Survey results revealed that, for Vanier scholars both in Canada and outside Canada at the time of application to the award, the most common ways they learned about the program were through colleagues/friends/word of mouth, from being approached by a graduate program coordinator or from being approached by their doctoral supervisor. Although the frequencies of these responses changed slightly between those inside and outside Canada, these findings imply that Vanier promotional activities, the Vanier CGS website and university organized events play a lesser or more indirect role in raising awareness of the program.

University Promotion of the Vanier CGS Program and its Recipients

Due to limited resources, few universities reported doing any promotion of the Vanier program, whether in Canada or internationally, other than on their website. International recruitment is generally limited, if the universities do any at all. The universities who do recruit abroad tend to target specific countries, and do not believe that the Vanier award has a place in that recruitment discourse (due to not being able to guarantee receipt of the award). In terms of Vanier scholars, universities often promote these individuals internally while some have had their recipients featured in the local news. One factor hindering the ability of universities to promote its scholars is the delay between when the decision is made (and the student is notified) and when the federal granting agencies make their official announcement. This delay results in the universities being too late in promoting the news of the Vanier awards.

DFATD’s Promotional Activities

DFATD representatives report mentioning the Vanier scholarship whenever possible in their recruitment efforts, especially when the organization does not have any country-specific scholarships to offer. DFATD has promoted the Vanier and Banting programs through its Edu-Canada initiative and has distributed Vanier program brochures for use in Canadian embassies and high commissions, as well as numerous tradeshows. However, representatives interviewed from DFATD stressed that their organization can only promote the Vanier award to a certain extent, given its exclusivity and its prestige; it cannot be used as a mass-attraction tool. The Vanier award is also included on their website, which is said to be a very popular website for foreign students.

Missions abroad sometimes hold information sessions on scholarships, and they mention the Vanier scholarship among the options available to prospective students. However, DFATD representatives indicated that the organization does not have the resources to undertake specific recruitment abroad. Moreover, their mandate is to promote Canada as a research destination, not the Vanier scholarship itself.

The Vanier CGS Program’s Promotional Activities

Data obtained from a recent Vanier CGS communications report highlighted the various promotional activities undertaken by the program in 2012. Social media was an important promotional component of the Vanier program as it disseminated information through its Twitter and Facebook accounts. As of January 2013, there were 839 followers on Twitter, and the program has 779 Facebook fans. Since January 2012, the program published 163 tweets.

Evaluation Question

1.2. Have Vanier CGS recipients received training that has led to their research, leadership and academic development?

Survey findings demonstrate that the majority of Vanier scholars are satisfied with their skill development and training opportunities related to their studies. In terms of development, the majority of Vanier scholars surveyed reported a moderate to significant improvement of research skills for most of the research-related activities assessed. Areas where improvement in skills did not occur or were slight at best pertained to collaborative research with the private/government sector/government/not-for-profit (37%), international research collaborations (26%) and leading research projects (20%). In terms of teaching skills, improvement rates were not as high as compared to research skills. In particular, between 13% and 14% of Vanier scholars reported showing no improvement in areas such as developing course materials, developing teaching methods and teaching a university course.

For personal and professional skill development, close to half of Vanier scholars reported showing slight to no improvement for financial management, business management and personal/non-academic pursuits. One-quarter of Vanier scholars indicated that they showed little to no improvement in leading/influencing others within the larger community. However, the lack of skill development in some areas may not necessarily be an indication of training gaps as 86% of Vanier scholars reported being satisfied to very satisfied with the development of their academic and non-academic skills during their doctoral degree.

With respect to involvement in service-related activities during their doctoral studies (Figure 2.4), organizing or participating in volunteer activities was the most widespread activity reported on with 9 out of 10 Vanier scholars indicating some level of involvement. Overall, the majority of Vanier scholars reported being involved to some degree in service-related activities to their academic and larger communities.

Figure 2.4 – % of Vanier Scholars Reporting Involvement in Service-Related Activities

Source: Survey of Vanier Scholars (N=559)

Figure 2.4: long description

Table 2.6 – Frequency of Interactions of Vanier Scholars with Researchers
% of Vanier Scholars Who Reported Interaction at Least Once a Month
Researchers in your discipline in Canada 79%
Researchers in your discipline outside Canada 51%
Researchers in other disciplines in Canada 57%
Researchers in other disciplines outside Canada 25%

Source: Survey of Vanier Scholars (N=578)

In terms of training opportunities available to Vanier scholars during their studies, the majority reported satisfaction with the opportunities they received to develop their skills in research, teaching, service to the academic and/or larger community and personal/professional. The degree of satisfaction was strongest with respect to the research and personal/professional opportunities they received. In terms of dissatisfaction, teaching opportunities garnered the highest level of discontent at 19%. Finally, 8 out of 10 Vanier scholars expressed some degree of satisfaction with their overall research environment.

Table 2.7 – Research-Related Publications and Products Produced by Vanier Scholars
Research-Related Publications/Products Average ± Std. Dev.
Articles written or co-written published or accepted in peer-review journals 4.5 ± 6.0
Research papers, books, book chapters and technical publications authored or co-authored published or accepted for publication 2.7 ± 6.7
Grey literature products written or co-written 1.1 ± 2.6
Oral or poster conference presentations given 9.2 ± 9.4
Oral or poster conference presentations at international conferences 4.5 ± 5.2
Art installations, productions or exhibitions produced 0.2 ± 1.2
Research tools produced 0.7 ± 2.0
Tools for research-related activities produced 0.6 ± 1.9
Patent applications filed 0.1 ± 0.6
Patents granted 0.3 ± 0.2
Other intellectual property claims filed 0.1 ± 0.4
Other intellectual property claims granted 0.0 ± 0.3

Source: Survey of Vanier Scholars (N=535)

With respect to the doctoral supervisors of Vanier scholars, survey results reveal that the majority have supervised one scholar (86%) up to this point and tend to be male (63%). In particular, supervisors of NSERC funded Vanier scholars are primarily male (82%) as compared to those supervising award recipients from the other two granting agencies (CIHR supervisors are 62% men while SSHRC supervisors are 59%). Additionally, Vanier scholar supervisors tend to be either a professor (63%) or associate professor (28%) with 23% having received a Canada Research Chair (1% were in receipt of a Canada Excellence Research Chair). Approximately one-third of Vanier scholars reported they had interacted with their supervisor(s) several times a week while only 11% indicated that they interacted once a month or less. Overall, 81% of Vanier scholars reported being satisfied or very satisfied with the supervision provided by their primary doctoral supervisor. In terms of interactions with researchers, close to 8 out of 10 Vanier scholars interacted at least once a month with Canadian researchers in the same discipline while only one quarter did so with researchers in other disciplines abroad (Table 2.6).

An important indicator of excellence achieved at the doctoral level is research productivity. Table 2.7 outlines the average number of research-related publications and products produced by Vanier scholars during the period of their doctoral studies. Productivity was greatest in terms of conference presentations given (including international presentations) and peer reviewed articles published or accepted. As well, approximately half of Vanier scholars report a new theory (53%), findings cited by others (47%) and a new research method (45%) had resulted from the research they have conducted during their doctoral studies (Figure 2.5).

Figure 2.5 – % of Vanier Scholars Reporting Research-Related Outcomes

Source: Survey of Vanier Scholars (N=498)

Figure 2.5: long description

Evaluation findings show that universities contributed greatly in terms of providing financial support to Vanier scholars during their studies. In terms of their success in receiving awards and prizes in addition to the Vanier CGS award, close to 9 out of 10 Vanier scholars reported having received awards/prizes from their university (Table 2.8). With respect to employment-related financial support received, 19% of Vanier scholars reported working in a non-academic position during their doctoral studies indicating that most award holders do not need to seek non-academic employment while in school. Furthermore, close to 6 out of 10 Vanier scholars reported having obtained a teaching assistantship while close to one-third reported having received at least one research assistantship from a federal granting agency (Table 2.9).

Table 2.8 – % of Vanier Scholars Who Received Awards/Prizes
Sources of Awards/Prizes % of Vanier Scholars who Received Award/Prize
Awards/prizes from their university 87%
Awards/prizes from a Canadian provincial body 38%
Awards/prizes from a Canadian federal granting agency (excluding the Vanier CGS award) 35%
Awards/prizes from a non-Canadian organization 20%
Awards/prizes from a not-for-profit organization 21%
Awards/prizes from a private sector 9%
Other 4%

Source: Survey of Vanier Scholars (N=581)

To contextualize these findings, data from Statistics Canada’s survey of earned doctorates (Statistics Canada, 2008) showed that the two most frequent sources of financial support obtained by doctoral students during their graduate studies (which includes Master’s and Doctoral studies) was a fellowship or scholarship from the student’s own institution (65%) and a teaching assistantship from the student’s own institution (65%). Data obtained in the evaluation indicates that Vanier scholars have a higher success rate than Canadian doctoral students in obtaining university scholarships/fellowships and a similar rate for teaching assistantships – which is another measure of their excellence at the doctoral level.

Table 2.9 – % of Vanier Scholars Who Received Employment-Related Financial Support
Sources of Employment-Related Financial Support % of Vanier Scholars who Received Support
Research stipend/assistantship paid from a CIHR Strategic Training Initiative in Health Research (STIHR) grant 4%
Research stipend/assistantship paid from an NSERC Collaborative Research and Training Experience (CREATE) grant 2%
Research stipend/assistantship paid from a research grant from a federal granting agency (CIHR, NSERC, SSHRC) 34%
Research stipend/assistantship paid from a different/unknown source 43%
Teaching assistantship 60%
Non-academic employment position 19%

Source: Survey of Vanier Scholars (N=645)

Evaluation findings indicate that the high dollar amount of the award results in a large majority of recipients not having financial debt related to their doctoral studies. For those Vanier scholars who have not yet completed their studies, 80% reported having no debt related to their doctoral program. For those who completed their doctoral studies, 90% reported having no debt related to their program upon graduation. This finding holds true even when comparing Canadian and foreign students (who can incur higher costs to study in Canada): 80% of foreign students who did not complete their studies were debt free (compared to 82% of Canadian students) while 92% of foreign graduates did not owe any money (compared to 90% for Canadians). These findings are consistent with data obtained through Statistics Canada’s 2005/06 survey of earned doctorates which showed that 88% of doctoral students did not have any debt related to their graduate studies (which also included costs related to their Master’s degrees) (Statistics Canada, 2008).

Impact of Vanier CGS on Universities

University representatives interviewed generally felt that the Vanier program had had little to no impact on the training offered at their institutions, given the limited number of awards distributed and received by each institution. As well, some felt it was difficult to assess attribution of changes to the university by the program since many award recipients are already enrolled prior to winning the scholarship. Some university representatives assumed that the Vanier program made a difference in the quality of the training offered at their institution, but they could not provide data supporting this idea. To some, the Vanier scholars did not make a greater difference than that made by CGS award holders. Others argued that having students of Vanier calibre at their institution inspired others and elevated the quality of the training.

However, evidence collected through the evaluation confirms the assumptions made by several university representatives interviewed. According to Vanier supervisors surveyed, award holders have had the biggest positive impact on their departments in terms of the quality of training provided (70%) and the quality of the research environment (67%). According to them, the aspect of their departments where Vanier scholars had the least positive impact was on the research equipment, instruments and facilities available (27%).

The presence of Vanier scholars at an institution was mentioned as a validation of the quality of the research conducted there: it is assumed that Vanier caliber students would have plenty of opportunities to study in various universities, but that when they select a Canadian institution and that their excellence is recognized by such a prestigious award, the quality of the research program is confirmed.

Evaluation Question

1.3. Have Vanier CGS recipients received training that meets their research and career objectives?

In terms of research and career objectives, Vanier scholars who had not yet completed their doctoral studies were asked what sector(s) they intended to work in after graduation. The majority (79%) indicated an intention to work in the university sector while close to one-third planned to work in the private sector (32%), government (37%) and not-for-profit (28%). As well, 7 out of 10 Vanier scholars indicated a moderate to high interest in pursuing a postdoctoral degree upon completion and 84% felt the same towards pursuing a position as a university professor/researcher. As to whether their doctoral training has met their needs, close to 8 out of 10 (78%) Vanier scholars noted that the training they received during their doctoral studies was moderately to extremely useful in preparing them for their career.

Approximately 38% of Vanier scholars reported some gaps in the training they received during their studies. The most often cited training gap was related to teaching skills. Because Vanier scholars are well supported financially due to the high dollar value of the award, it was felt by several award recipients that university departments may not allow or encourage award holders to receive teaching assistantships in order to ensure that other students have access to that funding. Without exposure to teaching and the skills required, several scholars reporting feeling unprepared for an academic career. The second most reported training gap related to the lack of opportunities to develop the skills required to conduct and manage a research project.

The CGS-Michael Smith Foreign Studies Supplement (CGS-MSFSS)

Vanier CGS award holders are eligible to receive a CGS Michael Smith Foreign Studies Supplement of up to $6,000 to study abroad for up to six months of their doctoral degree. A total of 14% of Vanier scholars surveyed indicated that they received the CGS-MSFSS (an almost equal distribution across Vanier scholars in health, natural sciences and engineering and social sciences and humanities). The majority of Vanier scholars reported that the CGS-MSFSS allowed them to accomplish their international research-related objectives, allowed them to establish relationships and networks within the country/countries they visited and enabled them to achieve outcomes they would not have achieved without the award. A little under three-quarters (68%) of Vanier MSFSS recipients indicated that the award amount was appropriate for their research needs and 71% confirmed that the award duration was appropriate.

Progress Towards Achievement of Intermediate Outcomes

Evaluation Question

2. To what extent has the Vanier CGS program progressed towards the achievement of its expected intermediate outcomes?

Retention of Canada’s Top Doctoral Students

According to results obtained through the survey of Vanier scholars, 20% of award holders have completed their doctoral studies with 59% of graduates having applied for the award in 2008 and 38% in 2009. Findings reveal that 63% of Vanier scholar graduates were living in Canada at time of survey and that students in natural sciences and engineering were more likely to leave at the end of their studies (Table 2.10). The country outside of Canada most frequently listed as a current residence for Vanier graduates was the United States (23%). Of those who left Canada and did not locate to the United States, 71% were residing in a European country.

These findings are in line with data on Canadian doctoral students in general as previous research has shown that slightly more than 12% of doctoral graduates end up living in the United States after graduation (Statistics Canada, 2011). As well, 83% of the graduates living in the United States intended to return to Canada. According to Statistics Canada (2008), 46% of Canadian doctoral graduates receive their post-doctoral training from an institution outside of Canada. Of the Vanier scholar graduates who are living outside of Canada, 61% indicated that they are employed as a postdoctoral fellow or associate – higher than the overall rate for Canadian doctoral graduates.

Table 2.10 – % of Vanier Scholar Graduates Retained
% of Vanier Scholar Graduates Who Stayed in Canada Post-Graduation
Foreign Students
(N=12)		 Canadian Students
(N=91)		 Total
(N=103)
Health Sciences 50% 77% 76%
Natural Sciences & Engineering 50% 49% 49%
Social Sciences & Humanities 25% 74% 67%
Vanier CGS Total 42% 66% 63%

Source: Survey of Vanier Scholars (N=103)

Of the Vanier graduates who left Canada, 8 out of 10 Canadian students indicated a moderate to high likelihood of returning to the country. The likelihood of foreign Vanier graduates returning to Canada was lower with 29% indicating a moderate to high likelihood of returning. Canadian Vanier graduates who left the country indicated that prospective employment (87%), an actual employment offer (81%) and research funding and equipment available (79%) were the most important factors that influenced their decision to leave. For foreign Vanier scholar graduates now living outside Canada, 86% indicated that prospective employment and an actual employment offer were the two most important factors in influencing their decision.

For Vanier scholars who have not yet completed their studies and who indicated a planned country of residence post-graduation, 95% of Canadians indicated an intention to remain in the country (compared to 86% for foreign students). Results from Statistics Canada’s survey of earned doctorates (Statistics Canada, 2008) showed that one fifth (21%) of Canadian doctoral students had planned to live outside Canada upon completion of their degree. Furthermore, for those intending to leave, the United States was the most frequently identified destination by both groups (9%) which is in line with results from previous research on doctoral students in Canada.

Over 90% of both Canadian and foreign students indicated that quality of life is the most important factor in their decision to stay in or leave Canada after they graduate. Location of spouse/family/friends (85%) and prospective employment (83%) were the next most important factors identified by Canadian Vanier award holders while prospective employment was listed as second for foreign students (88%) and research funding and equipment available was third (80%).

Vanier Scholar Post-Graduation Employment

Of the Vanier scholars who have completed their studies, 89% indicated that they are employed (83% full-time and 6% part-time). Employment rate post-degree is somewhat higher for Vanier scholars than the 85% employment rate (8% of which were part-time) reported by Canadian doctoral students living in Canada or the United States two years after their graduation in 2005 (Statistics Canada, 2011).

Table 2.11 – % of Vanier Scholars Employed by Sector
Sector % of Vanier Scholars Who Are Employed
University 71%
Private sector 11%
Government 11%
Not-for-profit 1%
University & Government 1%
Private Sector & Not-for-profit 1%
Private Sector & Government 1%
Private Sector & University 2%

Source: Survey of Vanier Scholars (N=91)

Approximately three quarters (75%) of Vanier scholars employed indicated that they are working in the university sector and 6% indicated being employed in two sectors (Table 2.11). Employment data on Canadian doctoral student graduates show similarities to findings obtained in this study (although Vanier scholars are more likely to obtain employment in universities than doctoral students in general): a little over half (56%) of Canadian doctoral graduates living in Canada and the United States two years after graduation in 2005 were employed in educational services with the vast majority of those in education working in a university (87%) (Statistics Canada, 2011).

A little over half (51%) of Vanier graduates who are working in the university sector are doing so as postdoctoral fellows/associates. When asked how many postdoctoral fellowships have they held, 48% indicated one postdoc, 6% reported holding two while 46% indicated that they have not held any. Approximately 17% are working as research faculty, scientists, associates or fellows and 15% are teaching faculty.

Table 2.12 – Annual Income Range for Vanier Scholar Graduates
Income Range % of Vanier Scholars Reporting Level of Income
$5,000-$9,999 1%
$15,000-$19,999 1%
$25,000-$34,999 1%
$35,000-$49,999 30%
$50,000-$74,999 35%
$75,000-$99,999 24%
$100,000-$149,999 4%
$150,000-$199,999 1%

Source: Survey of Vanier Scholars (N=91)

Overall, the majority of Vanier scholars working in non-academic sectors are employed as researchers and are earning an average annual income (before deductions) between $35,000 and $99,999 (Table 2.12). Statistics Canada data on Canadian doctoral graduates showed that the median income for all graduates two years after graduation in 2005 was $65,000 while graduates at the 25th percentile were paid $48,387 and those at the 75th percentile were paid $79,000 (Statistics Canada, 2011). In terms of the match between education and employment, 81% of Vanier graduates stated that their current position was moderately to perfectly related to their doctoral degree.

For those who indicated working in a job that is not somewhat related to their studies, the most frequently mentioned reasons were a change in career or professional interests, a job in their degree field was not available and the location of the job. In terms of exerting a moderate to high degree of leadership and influence through their primary occupation, 67% of Vanier scholars indicated they lead/influence others within their organization while a much smaller proportion (37%) stated that they lead/influence the strategic direction of their organization (although it should be noted that many graduates are employed in post-doctoral positions).

Enhancement of the Reputations of Canadian Universities

University representatives interviewed often indicated that the presence of Vanier recipients would enhance the reputation of their faculty, program or laboratory, but that it would have little to no impact on the reputation of the institution as a whole. It was noted that a university's reputation was built on numerous factors, and that prestigious scholarships were only one of those. The presence of Vanier scholars at an institution was mentioned however as a validation of the quality of the research conducted there: it is assumed that Vanier caliber students would have plenty of opportunities to study in various universities, but that when they select a Canadian institution and that their excellence is recognized by such a prestigious award, the quality of the research program is confirmed. As with the impact of the Vanier program on training and research, universities were not able to provide data on any impacts to their reputation.

Vanier scholar supervisors indicated that the program has had the greatest positive impact in terms of enhancing the reputation of their departments through enhancing their own reputation as a supervisor of doctoral students (69%). The area in which the highest proportion of supervisors (14%) identified as having not been impacted by the Vanier CGS program is the reputation of Canada as a research environment.

Two areas in which a minority of supervisors indicated the program has had a negative effect has been on the climate between doctoral students under their supervision (7%) and the climate within their research team or lab (6%). This issue was identified by several university representatives interviewed who noted that the high funding level provided by the Vanier program has, in some instances, reportedly created tension, uneasiness and sometimes envy, given the large differences in revenue between unfunded PhD students, granting agency-funded students and Vanier awardees who are working at the same level on the same problems.

Enhancement of the Capacity of Universities to Attract the Best and Brightest

To some university representatives interviewed, the Vanier scholarship has influenced their institution’s capacity to attract the best and brightest students because of the funding level provided by the program, which is much greater than what it could otherwise offer. However, university respondents also brought up that the number of Vanier awards was too small to be used as a recruitment tool: the selection of university nominees is very stringent, and the award is extremely competitive, so universities are unable make any guarantees to PhD students, whether prospective or already enrolled, about receiving the award. The role of the Vanier scholarship as a recruitment tool for Canadian and foreign students is furthermore limited by the reluctance of supervisors to nominate students with whom they have not yet worked with (as previously discussed under evaluation question 1.1).

Progress Towards Achievement of Long-Term Outcomes

Evaluation Question

3. To what extent has the Vanier CGS program progressed towards the achievement of its expected long-term outcomes?

Recognition of the Vanier CGS Program as a Prestigious Award

Evaluation findings confirm that award recipients and their supervisors view the Vanier CGS program as a highly prestigious award in Canada. Results are lower however when looking at the issue of its prestige internationally although progress is being made towards achieving international recognition with more foreign students rate the award as having more prestige worldwide than Canadians.

Survey results reveal that Vanier scholars, in rating the prestige of all federal granting agency masters and doctoral scholarships and training grants, viewed the Vanier award as having a very high level of prestige (90%) – the highest level of agreement for that rating compared to all other funding opportunities rated. A similar opinion was also shared by their doctoral supervisors the majority of whom felt the award carried a very high level of prestige (88%) in Canada.

With respect to the prestige of the Vanier scholarship amongst the doctoral scholarship landscape outside of Canada, ratings by Vanier scholars were somewhat lower with 27% indicating the award held very high prestige and 30% rating it as having high prestige. Approximately 42% of supervisors felt that the award carried a very high prestige internationally (while 30% felt it had high prestige). Foreign students rated the prestige of the Vanier award outside of Canada more favourably than Canadian students. Approximately 38% of foreign students rated the Vanier scholarship as having very high prestige abroad while 30% viewed it as having high prestige.

In terms of the effect of the prestige of the award on students, it was felt by university representatives interviewed that the Vanier program's reputation may be more relevant for students interested in a career in academia, whereas it was unclear what weight the award had on the reputation of a doctoral student who was headed for a career in consultation or public administration, for instance. This was linked to the perceived notion that the private, non-academic sector is less preoccupied with awards when assessing the merit of job applicants than universities would be.

Efficiency and Effectiveness - Program Design and Delivery

Evaluation Question

4. Has the Vanier CGS program been implemented as intended?

Levels of satisfaction with the application and selection process provide client perspectives on the efficacy of the delivery of the Vanier CGS program - both at the university and federal granting agency level. Overall, Vanier scholars reported a consistently high level of satisfaction and low level of dissatisfaction across several aspects of the process (Figure 2.6). Award recipients were most satisfied with the information available on how to apply to Vanier (91%), the eligibility requirements (89%) and the information available on the program (88%). The issues of program delivery that a minority of Vanier scholars were most unhappy with pertained to the clarity of the university selection process (15%), the amount of work required to complete an application (12%) and the application form itself (10%).

Figure 2.6 – Opinions of Vanier Scholars on the Application and Selection Process

Source: Survey of Vanier Scholars (N=519)

Figure 2.6: long description

In terms of the design of the Vanier CGS program, Vanier scholars surveyed indicated a high degree of overall satisfaction with the amount, duration and reputation of the award. Specifically, 92% of Vanier scholars expressed the highest level of satisfaction in regards to the award amount (with satisfaction levels consistent across fields of study). Results were somewhat lower for the reputation of the award with 83% indicated a high level of satisfaction (with Vanier scholars working in social sciences and humanities indicating a slightly lower degree of satisfaction).

Additionally, it was sometimes indicated by universities that the length of the Vanier funding was insufficient, given that in some disciplines it is rare for students to complete their doctoral studies in three years. As one university representative explained, "given that is it extremely uncommon for a doctoral student to complete a program in less than four years, some other institutions are offering four-year funding packages, which can put Vanier at a slight disadvantage."

Photo: 2013 Vanier Scholar Jennifer Shaw (SSHRC) – Simon Fraser University. Photo courtesy of Simon Fraser University.

Although only a small percentage of award holders expressed dissatisfaction with the length of the award, Vanier scholars did not express as high a level of satisfaction on this dimension (particularly students in social sciences and humanities) with 62% feeling very satisfied. This finding is not surprising given that the Vanier award covers three years of study and recent data shows that the average completion time for Canadian PhD students ranges from slightly under 15 terms – or five years, based on three terms per year – in the physical sciences and engineering, to a high of 18.25 terms, or just over six years, in the humanities. The mean time-to-completion was 15.4 terms in the health sciences and almost 17 terms in the social sciences (Tamburri, 2013). A suggestion to lengthen the duration of the award was made by several Vanier scholars when asked for ways to improve the program.

Although no key informant interviewees argued that the funding level for the award was too low, some argued that it was too high (regardless of the student's origin), especially in comparison to the salaries the postdoctoral fellows with whom the awardees work. It was suggested by interviewees that having more CGS scholars would be preferable to having very few Vanier scholars. It was sometimes perceived that going beyond the $35,000/year CGS doctoral award funding level had a lesser incremental impact on the student, whereas those funds would have more impact if they were used to fund more students at the CGS level. As well, they were concerned that it may make other students feel less worthy. The idea of turning CGS awards into "the silver medal" was viewed as detrimental in the long run. However, at the granting agency level, stakeholders felt that the funding amount was commensurate with their aspirations to attract the very best doctoral students and leaders world-wide to Canada.

Overall, university representatives were generally satisfied with the Vanier CGS program design and were supportive of the Vanier scholarship being open to international students, as this is their only national scholarship vehicle to attract such students. As one university representative noted, "universities don't have many resources to provide funding packages to students, and international students cost more than Canadian ones, so we now realize that without Vanier we wouldn't be able to get international students."

Photo: 2013 Vanier Scholar Zhihui Yi (NSERC) – École Polytechnique de Montréal. Photo courtesy of Natural Sciences and Engineering Research Council.

While some university representatives viewed the students' ability to apply to the Vanier program as well as other tri-agency scholarships simultaneously positively, it was sometimes viewed as a program flaw. To some, if a student was nominated for a Vanier (and was awarded it) but had already applied for a Canada Graduate Scholarship, for instance, it meant that their university had "wasted" a CGS application on that student.

From the perspective of Vanier scholars, the strengths of the program are the award amount (which many claimed allowed them to focus solely on their research, increase their productivity and pursue their own research that was more innovative and less traditional), the prestige of the award, the leadership component as well as international student eligibility.

The areas of suggested improvement (by order of most frequently cited) were:

According to Vanier scholar supervisors, the strengths of the program are that it provides adequate funding to doctoral students, supports excellence (according to many this is due to its leadership component) and is very prestigious. The most common area for improvement identified was that the Vanier award amount is too high. Specifically, it was mentioned that it would be in Canada’s best interest to award more grantees with less money. They noted that post-doctoral funding and salaries are much lower than the value of the Vanier award and can lead to disappointment or even discourage Vanier scholars from pursuing a post-doctoral position. As well, they claimed that the level of award creates a class-system amongst students in the lab where one person is earning almost double that of other students for completing the same work. Many also commented that the selection process for Vanier is relatively arbitrary as there is little actual difference between students at this level. Others cited that the high award amount can create conflict or negativity in the lab environment which affects productivity. Finally, Vanier scholar supervisors suggested that research and travel costs be allowable expenses under the Vanier scholarship.

Vanier scholars (78%) reported that the award had the strongest positive impact on their current financial situation (for those who have not yet graduated), the need for obtaining income during their studies (68%) and the time they were able to devote to their doctoral studies (66%). In terms of influencing the pace of their progress in their doctoral program, 77% indicated the award had an overall positive impact.

Vanier CGS Nomination Allocation System

Over the period of the first Vanier CGS allocation cycle (2010–2012), a total of 52 Canadian universities received at least one nomination allocation. At the end of the cycle, only 21% of institutions fulfilled their allocation quota. Of the 79% whose quota was unfulfilled, 8% did not submit a single nomination. G-13 institutions received the bulk of the allocations (71%) and provided nominations for 88% of their allocations (compared to non-G-13 universities who used 77% of their allocations).

Photo: 2013 Vanier Scholar Mohamed Soliman (CIHR) – University of Toronto. Photo courtesy of the Lunenfeld-Tenenbaum Research Institute, University of Toronto.

In terms of how institutions manage their allocations over the cycle, university representatives interviewed noted that earmarking nominations for each year was one way of planning for the funding cycle, though the earmarked allocations were followed loosely. In some instances, more nominations were submitted in the last year of the cycle, knowing that their students may not meet all the criteria, in order to use up their allocations. A review of Vanier CGS administrative data confirmed that the largest number of nominations submitted occurred in the final year of the first allocation cycle (Figure 2.2).

Key informant interviewees were generally at ease with the university allocation system, because it forces the universities as a whole to work harder to develop research capacity and obtain tri-agency funding. In this sense, the allocation system is viewed as stimulating; however, it was also mentioned that smaller institutions had more difficulty operating within this allocation system because of their limited resources and small quotas.

Some university representatives thought that the three-year allocation period allowed the universities to be flexible in their nominations, while highlighting the exclusivity and prestige of the scholarship. However, it was also indicated that in some cases universities would be more comfortable with a yearly allocation quota, because it would limit the chances of passing up on a high-calibre candidate early on in the funding cycle.

The idea was also brought up that the allocation system was skewed towards maintaining status quo in terms of resource allocations. Because the indicators used to determine the allocations are internal to the three granting agencies, there is little room for change. As one interviewee explained, "the rich get richer: it doesn't foster a growth or changes in the landscape of research training in Canada." On the other hand, the current indicators were underscored as being reliable, since the data were gathered by the granting agencies themselves.

It was in some cases suggested that taking the federal granting agencies' $20,000 doctoral student awards into consideration would improve the allocation system; it is possible that this suggestion was made in part because it would be at these universities' advantage. It was also suggested that the universities' success rate in the granting agency doctoral scholarship competition be taken into consideration, along with the number of doctoral students enrolled at each university or the timeliness of doctoral graduation.

Regarding the possibility of allocations being reserved specifically for foreign students, interviewees were sometimes in disagreement with the idea, indicating that the Vanier scholarship should reward excellence, not origin. It was often mentioned that cultural differences and discrepancies in opportunities rendered the leadership criteria more difficult to judge in foreign students (see below). Some universities spoke against quotas for international students because they feared that they would not be able to fill those quotas; this was especially true of universities attracting few international students. The scarcity in student funding was also an argument used against quotas for international students.

When university representatives spoke favourably of implementing quotas for international students, their argument was based on the principle that the knowledge economy transcends borders. Moreover, some Ontario-based universities explained that their provincial funding does not cover international students, and that having a portion of the Vanier scholarships reserved for international students would be a positive decision given their situation.

The equal split of the candidates among the granting agencies was generally viewed as appropriate; indeed, some were loath to favor one agency over the others. They also thought that having an uneven split would put interdisciplinary research at a disadvantage. In that regard, the suggestion was made that a portion of the scholarships be reserved outside of the granting agency envelopes for Vanier scholars conducting interdisciplinary research. On the other hand, some university representatives argued that given that NSERC and CIHR have more funding than SSHRC; it would make sense to allocate more grants to those agencies. At the same time it was also recognized that the higher demand for SSHRC funding should be taken into consideration.

Vanier CGS Nomination Selection Process

Faculty members, potential supervisors and graduate program coordinators were often relied on to identify students among those applying for scholarships as potential Vanier calibre, as they are closer to them. In some universities, it is possible for potential candidates to self-identify, in which case students need to submit documentation. Universities noted that challenges to identifying potential nominees included not being able to identify students of a high-enough calibre, students turning down invitations to apply due to the additional work required over the CGS award (and the low odds of being awarded) as well as some identified candidates not being eligible because they had received a CGS award in their first year of doctoral studies.

Photo: 2014 University of Toronto Vanier Awards Announcement – Toronto, Ontario. Photo courtesy of the University of Toronto.

Universities each follow their own process in selecting Vanier nominees and a wide range of processes are used to support candidates once they had been identified. At the university level, applications from Canadian and international candidates are reviewed together by an internal adjudication committee usually comprised of faculty members (which is sometimes the same committee as for the other tri-agency scholarships) and assessments are based on the Vanier selection criteria with leadership being emphasized. Some interviewees explained that among equally excellent academic dossiers, the leadership component would be the factor that would discriminate among candidates.

However, some university representatives indicated that their internal committees found the leadership criterion difficult to evaluate because they had less experience gauging an applicant's leadership (as opposed to academic excellence for instance, which they are used to assessing); the leadership criterion was perceived as rather subjective. It was suggested that more details be provided around the definition and operationalization of the concept (as well as further insight into the Vanier CGS selection criteria). To this end, some suggested that the universities should receive feedback on their nominations from the Federal Granting Agencies’ selection committees and felt that the Vanier program had a sense of secrecy to it.

Evaluating candidates' leadership skills also proved challenging for universities when it came to international students, because they didn't want to assume that international students’ cultures and governments did not offer the same opportunities for exercising leadership as does Canada. This was noted to be a factor in potentially limiting access to female foreign students in the Vanier program. Universities were generally struggling with the assessment of leadership in both Canadian and foreign students but particularly the latter, and it was suggested that the leadership criterion should be adapted for international students.

Program Delivery at the Granting Agency Level

Photo: 2013 Université Laval Vanier Awards Announcement – Quebec City, Quebec. Photo courtesy of the Vanier Banting Secretariat.

Universities expressed concerns pertaining to the timing of the Vanier application process. For one, the deadline was thought to be too close to the start of the fall semester, which is very busy for faculty members and administrative personnel, and leaves very little time for students to prepare their applications and for the university adjudication committee to review them and provide feedback. Moreover, the time lapse between the application period and when the decision is made at the agency level discourages universities from presenting first-year doctoral nominees, because they would have had to apply to the Vanier program almost a full year before starting their doctoral studies. This proved especially challenging when seeking to attract new and foreign students to a university.

It was also noted by current and former Vanier granting agency selection committee members that the Vanier CGS program’s application deadline did not work in favour of international applicants given internal institutional deadlines for graduate program admissions. In addition, concern was expressed that current timelines for application submission and award decision might be potentially limiting the applicant pool, given that a number of potential Vanier CGS recipients will be taking up other awards. The level of uncertainty and risk that universities had to manage when nominating new students was identified as the main challenge faced in the delivery of the program at the university level. Some universities also mentioned that the tardiness of the announcement was problematic for international students, because it did not leave them enough time to make travel and visa arrangements.

Vanier CGS selection committee members who participated in an expert panel raised the suggestion of a direct applicant route for international applicants who might not come to Canada without the scholarship and who are applying for awards in other countries around the world. Selection committee members agreed that significant changes to the Vanier CGS program design would have to be considered to allow for a direct application stream for international applicants in order to increase the number of international applications (and their quality). As well, it was also suggested that allocation of a portion of the applications quota and funds be considered for the submission of direct international applicants.

In terms of difficulties encountered in the selection process at the Federal Granting Agency level, committee reviewers agreed that the leadership criterion was the most challenging and contentious, and posed the greatest variance when reviewing applications during the expert review panel exercise convened as part of the evaluation. On this note, panel members agreed that the assessment of academic excellence and research potential were more straightforward but that leadership generated extensive discussions during Vanier selection committee meetings. They noted that leadership is fundamental to the Vanier CGS program and, as was reported by university representatives regarding their nomination selection process, is often used in the selection process to differentiate between candidates. It was noted that on Vanier CGS selection committees, leadership has been interpreted in different ways, however, the panel discussions let to a degree of consensus in that it should be assessed with the same sense of quality (in other words ‘world-class’) as the other selection criteria. Lastly, panel members agreed that leadership was fundamental to the Vanier CGS program.

However, although panel members agreed that the leadership criterion clearly distinguished the Vanier CGS from regular NSERC, CIHR or SSHRC doctoral awards, they also felt strongly that academic excellence and research potential were prerequisites of a Vanier CGS scholarship. On this note, panel members stressed that equal weight should be assigned to the academic, research and leadership potential criterions and that they be based on excellence/world-class. Furthermore, they recommended that applications with a score of three or below for any of the selection criteria be triaged out of the selection process, provided consensus is obtained from both reviewers.

Several suggestions were made by the Vanier evaluation expert panel on how to improve the assessment of the leadership criterion. They noted that in the context of a Rhodes scholarship, leadership is assessed based on initiatives that go above and beyond the actual and expected demands of a student. For example, some kind of initiative that brings good to the world or some kind of vision not directly connected to their studies (a leadership service component which is separate from the academic component). This could refer, for example, to a student excelling in music who then starts up an orchestra. They suggested that Vanier CGS applicants be required to clearly identify their key leadership accomplishments amongst those presented in the application, along with a brief rationale.

Panel members unanimously agreed on the need for a clearer tri-agency definition of leadership, noting that providing some good examples of what should be considered as strong leadership would be particularly useful such as making a distinction between leading an activity and being a participant.

Overall, most universities were satisfied with their interaction with the granting agencies and were generally pleased with the harmonization of the program because it lightened their administrative workload and spoke positively of the Vanier CGS common website. In terms of the Vanier Banting secretariat, not all universities had interacted with the group so they had less feedback to provide. Granting agency representatives viewed the Vanier Banting Secretariat as extremely important to optimal program delivery and viewed it positively because it allowed the three agencies to share a common vision and to collaborate more effectively. One downside noted was the risk of losing the flexibility to adapt the Vanier program to each agency, but this risk was thought to be limited and manageable.

Both the university and agency representatives unequivocally saw a value in having the funding competition managed centrally: centrality provides Vanier a prestige that it would not have if the universities managed the program independently. Federal Granting Agency staff recognized that the program was still young, and that improvements were still to be made.

Alternative Delivery Mechanisms

University representatives rarely had any suggestions for alternate or more effective program delivery mechanisms (aside from those already identified in other sections in the report). In terms of improvements, a one-year scholarship was suggested in addition to the current Vanier scholarships, in order to foster more international mobility; it was suggested that this increased support to foreign students wishing to come to Canada for a portion of their PhD would be beneficial.

With regards to the ability to use the Vanier award as a recruitment tool, it was suggested that the quota system be redesigned to provide universities with a given number of "guaranteed" awards for each agency. This would decrease the uncertainty of the award, therefore making it a more effective recruitment tool, in particular for international students. In the opinion of DFATD's representatives, the Vanier scholarship should be more effectively branded and potentially renamed to make the Canadian origin more prominent.

Vanier CGS selection committee members made several suggestions in terms of improving the delivery of the program (in addition to those previously mentioned in other sections of the report). First, it was agreed that some level of reporting (progress and/or final reports) should be requested from the Vanier recipients and their supervisors. They note that these reports would gather information on what has been accomplished during and at the end of the period of tenure of the award (It should be noted that an end of award survey for Vanier scholars was in development at the time of the evaluation). Selection committee members also suggested that universities should be persuaded to submit customized letters providing comments on the applicant, the quality of the infrastructure and the proposed supervisor and stressed the usefulness of personalized letters as part of the selection process. It was felt that this change would benefit the applicant, given that both the supervisor and the institution directly benefit from Vanier CGS funding. It was also proposed that the Vanier Banting Secretariat make the necessary refinements to their instructions for university nomination letters, and provide examples of excellent letters to institutions. In order to reduce workload for Vanier CGS selection committee members in assessing research potential, it was suggested that applicants be asked to provide the standard impact factor for publication journals, along with a sample of articles to be included with their application form.

The issue of the grade point average (GPA) assessment was also raised by expert panel members in terms of assessing the academic excellence of Vanier candidates, noting that some applicants with low GPA but with awards and distinctions would ultimately receive a medium score for academic excellence. Members noted that the current review criteria used by the Vanier CGS selection committees differed from the nomination criteria provided to universities. As such, instructions given to the Vanier CGS selection committees in terms of judging the GPA was based on the entire academic record, whereas universities were instructed to consider only the last two years of undergraduate studies. Panel members agreed that it would be useful to give universities the same guidance as selection committees.

Vanier CGS Alumni Network

Photo: 2012 Queen’s University Vanier Awards Announcement – Kingston, Ontario. Photo courtesy of the Vanier Banting Secretariat.

According to a performance measurement strategy developed for the Vanier CGS program in 2010, a Vanier Alumni Network was to be established during the second phase of the program with the purpose of ensuring that Vanier students are globally connected and have access to research collaboration opportunities in Canada and with international partners during their doctoral studies. At the time of the evaluation, an alumni network had not yet been implemented.

To help inform decision-making regarding the network, the evaluation sought the opinions of award holders in terms of their preferences for communicating with each other and with the Vanier secretariat. Results indicate that Vanier scholars showed a moderate to high interest in communicating with each other via email (57%), followed by a LinkedIn group (50%) and Facebook (46%). About 1 in 4 Vanier scholars provided their own suggestion with the majority indicating a desire to meet with other scholars in person through annual meetings and conferences. In terms of being kept informed of Vanier CGS-related updates and activities, periodic emails sent by Vanier program staff was preferred by most (74%) followed by a Vanier electronic newsletter (71%).

Evaluation Question

5. Has the Vanier CGS program been delivered by the federal granting agencies in a cost efficient

Findings show that the Vanier CGS program has been delivered by the federal granting agencies in a cost efficient manner. Data shows that the program has reduced its administrative costs as a proportion of its total expenses over time - both at the aggregate program level and at the level of the Federal Granting Agencies (Tables 2.13 and 2.15). As well, the program’s costs are below that of the total for the federal granting agencies over the period of 2009-10 to 2012-13 (Table 2.14). It should be noted that costs outlined in the tables below refer to the direct costs of the program incurred by the federal granting agencies (specifically the divisions administering the program). Also included in the costs are PWGSC’s contributions to rental space (located under “accommodations costs”) as well as branding expenses paid by SSHRC’s communication division. Costs associated with the administration of the Federal Granting Agency’s selection committees are included under direct operations and maintenance in Table 2.13. For the most part, staff time and salaries were derived based on estimates given by program management/staff.

It is important to note that several aspects related to the implementation of the Vanier program have impacted on its administrative costs. First, a Vanier Banting secretariat was created in October 2012 to provide administrative oversight of both programs across the federal granting agencies. Prior to the secretariat, the Vanier and Banting programs were supported by staff housed at CIHR, SSHRC and NSERC with the total number of full-time or part-time staff exceeding 20 individuals. In contrast, the secretariat, which is housed at CIHR, involves a total of 9 staff (5 from CIHR, 2 from NSERC and 2 from SSHRC). Furthermore, in the Fall of 2012 the Vanier Banting Steering Committee made the decision to dissolve the Vanier Selection BoardFootnote 8. The decision to dissolve the board was made to improve harmonization of the program and increase efficiencies. Finally, the absence of a planned Vanier Alumni Network has influenced the degree of cost associated with the program’s communications activities.

Table 2.13 – Vanier CGS Expenditures 2009-10 to 2013-14
2009-10 2010-11 2011-12 2012-13 2013-14
Vanier CGS Program Expenditures
Total Vanier CGS Admin Costs $758,476 $878,635 $808,746 $554,658 $564,171
CIHR Vanier Admin Costs $282,259 $365,933 $226,168 $240,231 $230,348
NSERC Vanier Admin Costs $169,494 $216,886 $250,297 $165,462 $176,916
SSHRC Vanier Admin Costs $306,723 $295,816 $332,281 $148,965 $156,906
Total Vanier CGS Awards Expenditures $7,970,833 $16,525,003 $23,988,078 $24,802,824 $24,807,499
CIHR Vanier Awards Expenditures $2,670,833 $5,500,000 $7,722,401 $8,286,264 $8,245,833
NSERC Awards Expenditures $2,700,000 $5,533,336 $8,047,076 $8,225,092 $8,275,000
SSHRC Vanier Awards Expenditures $2,600,000 $5,491,667 $8,218,601 $8,291,468 $8,286,666
Total Vanier CGS Total Expenditures $8,729,309 $17,403,638 $24,796,824 $25,357,482 $25,371,670
CIHR Vanier Total Expenditures $2,953,092 $5,865,933 $7,948,568 $8,526,495 $8,476,182
NSERC Total Expenditures $2,869,494 $5,750,222 $8,297,373 $8,390,554 $8,451,916
SSHRC Vanier Total Expenditures $2,906,723 $5,787,483 $8,550,882 $8,440,433 $8,443,572
Vanier CGS Admin Expenditure As % of Total Expenditures 8.7% 5.0% 3.3% 2.2% 2.2%
CIHR Vanier Admin Expenditure As % of Total Expenditures 9.6% 6.2% 2.8% 2.8% 2.7%
NSERC Vanier Admin Expenditure As % of Total Expenditures 5.9% 3.8% 3.0% 2.0% 2.1%
SSHRC Vanier Admin Expenditure As % of Total Expenditures 10.6% 5.1% 3.9% 1.8% 1.9%

Source: Federal Granting Agency Administrative Data

Table 2.14 – Federal Granting Agency and Vanier Operations and Expenses 2009-10 to 2012-13
2009-10 to 2012-13
Total Operations Total Expenses Total Operations / Total Expenses
CIHR $252,000,000 $4,058,400,000 6.2%
NSERC $222,554,000 $4,306,478,000 5.2%
SSHRC $121,591,000 $2,773,616,000 4.4%
Vanier CGS Total $3,000,515 $76,287,253 3.9%

Source: Federal Granting Agency Financial Data

Table 2.15 – Vanier CGS Expenditures by Federal Granting Agency 2009-10 to 2013-14
2009-10 2010-11 2011-12 2012-13 2013-14
CIHR - Vanier CGS Program Expenditures
Total Admin Costs $282,259 $365,933 $226,168 $240,231 $230,348
Direct Salary $188,478 $237,101 $150,864 $162,296 $154,010
Direct Operations and Maintenance $31,584 $50,589 $25,519 $24,378 $25,514
Employee Benefit Plan (20%) $37,696 $47,420 $30,173 $32,459 $30,802
Accommodation (13%) $24,502 $30,823 $19,612 $21,098 $20,021
Total Awards Expenditures $2,670,833 $5,500,000 $7,722,401 $8,286,264 $8,245,833
Total Expenditures $2,953,092 $5,865,933 $7,948,568 $8,526,495 $8,476,182
Admin Expenditure as % of Total Expenditures 9.6% 6.2% 2.8% 2.8% 2.7%
NSERC - Vanier CGS Program Expenditures
Total Admin Costs $169,494 $216,886 $250,297 $165,462 $176,916
Direct Salary $126,270 $148,073 $168,752 $105,421 $109,661
Direct Operations and Maintenance $1,555 $19,949 $25,857 $25,252 $31,067
Employee Benefit Plan (20%) $25,254 $29,615 $33,750 $21,084 $21,932
Accommodation (13%) $16,415 $19,249 $21,938 $13,705 $14,256
Total Awards Expenditures $2,700,000 $5,533,336 $8,047,076 $8,225,092 $8,275,000
Total Expenditures $2,869,494 $5,750,222 $8,297,373 $8,390,554 $8,451,916
Admin Expenditure as % of Total Expenditures 5.9% 3.8% 3.0% 2.0% 2.1%
SSHRC - Vanier CGS Program Expenditures
Total Admin Costs $306,723 $295,816 $332,281 $148,965 $156,906
Direct Salary $85,500 $101,594 $219,806 $111,493 $115,328
Direct Operations and Maintenance $193,008 $160,696 $39,939 $679 $3,520
Employee Benefit Plan (20%) $17,100 $20,319 $43,961 $22,299 $23,066
Accommodation (13%) $11,115 $13,207 $28,575 $14,494 $14,993
Total Awards Expenditures $2,600,000 $5,491,667 $8,218,601 $8,291,468 $8,286,666
Total Expenditures $2,906,723 $5,787,483 $8,550,882 $8,440,433 $8,443,572
Admin Expenditure as % of Total Expenditures 10.6% 5.1% 3.9% 1.8% 1.9%

Source: Federal Granting Agency Financial and Administrative Data

Conclusions and Recommendations

Conclusions

Evaluation findings indicate that the program is supporting world-class doctoral students – the majority of whom are Canadian, with most Vanier scholars demonstrating exceptional leadership in the area of research. The Vanier CGS award is having the strongest positive impact on students’ financial situation, their need for obtaining income during their studies and the time they were able to devote to their education.

The evaluation found that the Vanier CGS program is considered to be a highly prestigious award in Canada and the presence of Vanier scholars enhances the reputation of a faculty, program or laboratory but has little to no impact on the reputation of the institution. The majority of Vanier scholars are satisfied with their training and skill development and most report that their training has been useful in preparing them for their career. Of the Vanier scholars who have completed their studies, the majority are employed in the university sector and are living in Canada.

Findings from the evaluation regarding the communication preferences of Vanier scholars can help inform the development of an alumni network of Vanier scholars, which is an expected output of the program that has not yet been implemented.

The evaluation identified key barriers that negatively impact its ability to attract and recruit students from outside of Canada: the inability of institutions to guarantee receipt of the award, the high competitiveness of the program in relation to the relatively small number of scholarships awarded and the reluctance of doctoral supervisors to nominate students with whom they have not yet worked with and who are not already enrolled in the institution in which they are seeking Vanier support.

Opinions on the design and delivery of the program were positive overall, with the majority of Vanier scholars reporting satisfaction with the award amount and duration, however the evaluation found that the three year nomination allocation system is contributing to the reluctance of institutions to nominate new students (including foreign candidates) which limits the ability of the program to attract students from outside the country and has had a negative impact on the quality of candidates put forward for nomination. As a result, many universities suggested that the three year allocation cycle be changed to annual nominations.

As the objective of the Vanier CGS program is to be an internationally competitive scholarship program that enables Canada to attract world-class doctoral students (a key feature which distinguishes it from other federal scholarship programs) and given the reluctance of institutions to nominate foreign candidates, it is important that a portion of allocations be targeted to international students outside of the country in order to facilitate their nomination and better enable the program to meet its objective in terms of attraction. As well, the program’s competition deadlines were viewed by many as being problematic in terms of attracting new (including foreign) students to Canadian institutions and, as such, should be reconsidered.

The interpretation and assessment of the leadership criterion was identified as a challenge and, in light of the program’s expected long-term outcome that Vanier scholars become leaders in Canada and abroad, there is a need to further develop and clarify the definition and assessment of leadership given the important role it plays in the selection process at the university and federal granting agencies.

While the award amount is meeting the educational-related financial needs of almost all recipients, questions remain as to the incremental outcomes that result from the additional financial value of the Vanier award in comparison to other doctoral scholarship awards given the absence of comparator data at the time of this study. There is a need therefore to pursue an additional analysis using comparator data obtained through the evaluations of other doctoral award programs to identify the incremental outcomes that result from the additional financial support offered through the Vanier scholarship.

Recommendations

  1. To enable the Vanier CGS program to better meet its objective of attracting and recruiting world-class doctoral students to Canadian universities, the following changes to the allocation and application processes should be considered:
    1. The three year allocation cycle should be changed to annual allocations with restrictions placed on carrying forward unused quotas.
    2. A portion of nomination allocations should be targeted for foreign students not already enrolled in the institution in which they are seeking Vanier support, with the amount calculated, at least in part, on an institution’s international student enrollment rate. The Vanier CGS program should monitor the extent to which the target of foreign student nominees is being met, per institution and in total, after each competition and use this information as part of the calculation to determine targets and allocations for the next competition year.
    3. The timing of the application deadlines for the Vanier CGS program should be re-considered in light of findings from the evaluation.
  2. To help improve the assessment of the leadership criterion in the University and Federal Granting Agency selection processes and increase the extent to which it is interpreted consistently, Vanier CGS program management should establish a clearer definition of what leadership is and how it should be evaluated – especially for foreign students. Suggestions for improvement obtained through the evaluation should be taken into consideration and any changes made should be communicated to universities and Federal Granting Agency selection committee members.
  3. To help inform the implementation of a Vanier Alumni Network, data obtained through the evaluation on the communication preferences of Vanier scholars for interaction with other award holders and program staff should be considered to ensure that Vanier CGS graduates are connected to the program and to other scholars after completion of their studies.
  4. Evaluation findings demonstrate that the Vanier program is meeting the educational-related financial needs of almost all of its recipients and that it is viewed as a highly prestigious award in Canada. However, the evaluation was unable to assess what incremental outcomes are associated with the higher value of the scholarship in comparison to the CGS and Federal Granting Agency doctoral awards as evaluations of those programs were in progress at the time of this study and comparison data was not yet available. The surveys used in these scholarship evaluations were designed to enable a comparative analysis of data across programs. As such, it will be important to undertake a further analysis to assess the similarities and differences in the results achieved across programs as it can provide insight into what effect features of the Vanier scholarship, most importantly its award amount, have on outcomes. Issues to be examined would include incentives to enroll in studies, financial situation of students, training (including skill acquisition), research productivity, retention rates and employment. Based on this, it is recommended that Vanier CGS program management consider the results of this analysis in the future planning and design of the program.

Methodology

Consistent with Treasury Board guidance and recognized best practice in evaluation, a range of methods was used to triangulate evaluation findings. The approach of using multiple methodologies involving both quantitative and qualitative evidence is designed to ensure that the evaluation findings are robust and credible and that valid conclusions can be drawn about the performance of the program.

Limitations

Due to issues related to timing and resource requirements, several planned lines of evidence or analysis as detailed in the evaluation matrix of questions and indicators (see Appendix section) were not pursued as part of this study including a survey of Vanier applicants who were unsuccessful in receiving the award. These individuals will be surveyed as part of the ongoing evaluation of the CGS program. Furthermore, comparison data captured through evaluations of the CGS and Federal Granting Agency scholarship programs was not available for use in this study at the time of reporting as data collection for those evaluations had yet to commence. As part of the evaluation recommendations, it is suggested that this line of analysis be pursued in order to assess the similarities and differences in the results achieved across programs as it can provide insight into what effect features of the Vanier scholarship, most importantly its award amount, have on outcomes.

Administrative Data Analysis

A review of Vanier CGS program records including application and grant files and federal granting agency administrative data provided information that described and contextualized the program, its target audience and environment as well as aided in the assessment of program performance and relevance. Where appropriate, data was validated with Vanier CGS program staff to ensure accuracy and reliability.

Document Review

A document and data review was conducted to provide evidence that helped address several evaluation questions pertaining to program performance and relevance. Documentation reviewed included Vanier CGS program literature and reports, federal granting agency publications including previous evaluation reports, Government of Canada publications and documents as well as research articles - particularly those related to PhD education and doctoral students. A google keyword search as well as queries through online journal databases such as JSTOR was performed in order to identify and obtain documents focusing on identified best practices of research training. The search was conducted using key words such as “graduate student” and “funding.” All data collected through the document review was entered into Nvivo for analysis. In the case where electronic versions of information were unavailable, detailed notes were created and imported into an Nvivo file. Imported data was then coded according to relevant evaluation questions and indicators as detailed in the matrix of evaluation questions and indicators. Findings were then synthesized into a technical report following the same structure.

Expert Panel Review

SSHRC and NSERC evaluation staff, in consultation with Vanier-Banting Secretariat staff, convened a panel of current and former Vanier selection committee members to examine a purposively selected sample (N=75) of Vanier CGS nominees and recipients for years 2008–2012, with a view to assessing its overall quality (i.e., world-class). The Expert Panel consisted of three sub-committees who were representative of the disciplines and fields of study - health, social sciences and humanities and natural sciences and engineering. Vanier application files were selected for review by SSHRC/NSERC evaluation staff and divided equally among panel members (15 files per member), taking into account the following selection criteria: (i) gender; (ii) citizenship status (i.e., Canadian; foreign); (iii) funding status (recipient; non-recipient); (iv) language (French; English); and (v) competition year (within the years 2008 and 2012). Findings from the expert panel assessment were used to help address performance in terms of the program’s expected immediate outcome of attracting and recruiting world-class students by assessing the quality of Vanier nominees and recipients and determining whether the quality of applications has increased over time as well as examine issues related to program design and delivery.

Key Informant Interviews – University Representatives, Vanier CGS Program Staff and Federal Granting Agency and Government Representatives

A total of 31 interviews with N=36 key informants were conducted by Circum Network Inc. during the period of March 1st to April 8th 2013 with university representatives (Deans of Graduate Education and scholarship liaison officers), government officials, agency representatives, and Vanier CGS Program personnel. Qualitative data obtained from university representatives was used to help address several evaluation issues and questions as well as provide insight into the university review process including the criteria used to assess applications.

Whereas all relevant government officials, agency representatives and Vanier CGS Program personnel were interviewed, universities were selected using a stratified random sampling approach based on several criteria including (i) Vanier scholarship allocation and application data by agency; (ii) location in Canada; (iii) university enrolment; (iv) number of graduate programs; and (v) the existence of a medical school. The Evaluation Working Group, working in conjunction with the Vanier CGS secretariat, developed a list of potential interviewees from the Vanier eligible universities, the granting agencies and government departments. A total of 23 Canadian universities were included as part of the interviews, representing approximately 44% of all Vanier eligible institutions under the 2010–2012 allocation period. New universities under the new 2013–2015 allocation period were not considered as potential interviewees, given their limited contact with the program.

Table 4.1 – Interview Participation by Target Group
Interview Target Number of Interviewees Number of Interviews
Vanier Program Representatives 3 3
DFATD 3 1
Federal Granting Agencies 4 4
Canadian Universities 26 23
Total 36 31

Source: Vanier CGS Key Informant Interviews Technical Report

Organizational Scan

Findings from an organizational scan of N=17 national and international doctoral and postdoctoral scholarship programs carried out by Vanier CGS program staff was used to compare the design and implementation (e.g., program objectives, award value and duration, selection process, funding competition statistics) of the Vanier CGS program with similar national and international scholarship programs as well as identify best practices. In addition, CIHR Evaluation Unit staff undertook an organizational scan of scholarship programs similar to the Vanier CGS award to address issues and questions related to relevance including the identification of best practices and the degree of overlap of the objectives of Vanier CGS with other federal granting agency doctoral student programs. An internet search was conducted to identify similar scholarship awards worldwide to the Vanier program. A program was included as part of the scan if it had at least one similarity to Vanier in terms of target audience, prestige, award value, award duration, program objectives and/or selection process. A total of N=15 scholarship programs worldwide were included as part of the CIHR Evaluation Unit organizational scan. Invitations for follow-up interviews were sent to representatives from seven organizations whose programs were included. A total of three organizations were interviewed and provided additional data.

Online Survey – Vanier Scholars

The CIHR Evaluation Unit launched an online survey of Vanier scholarship recipients (N=830) over the period of February to March 2014 that provided quantitative and qualitative data that helped address a range of evaluation issues and questions. A census approach was undertaken to survey recipients awarded during the program’s first five years (2008–2012). Where appropriate, common survey questions were developed for use in the current evaluations of the Canada Graduate Scholarship (CGS) and agency specific Master’s and Doctoral scholarships to enable comparison of results across programs. Where appropriate, survey questions were also matched to those adopted in the first evaluation of the CGS program to allow for a comparison of results over time with doctoral students surveyed as part of that study. A survey of unsuccessful Vanier applicants (N=3,441) was originally planned as part of this evaluation. However, due to the close timing of other evaluations of federal granting agency scholarship programs, Vanier non-recipients who did not receive any type of federal scholarship support will be surveyed as unsuccessful scholarship applicants through the second evaluation of the CGS program. Vanier unsuccessful applicants who received other granting agency scholarships will be surveyed through the evaluation of CGS as a scholarship recipient for their most recent award held.

Table 4.2 – Vanier Scholar Survey Response Rate
Population Size Invitations Sent Completes Incompletes # Terminated Response Rate Margin of Error (95% CL)
CIHR Vanier Scholars 280 280 175 39 2 63% 4.5%
NSERC Vanier Scholars 277 277 172 48 0 62% 4.6%
SSHRC Vanier Scholars 273 273 174 35 0 64% 4.5%
Vanier CGS Scholars Total 830 830 521 122 2 63% 2.6%

Source: ResearchNet; Survey of Vanier Scholars

Online Survey – Vanier CGS Scholar Supervisors

The CIHR Evaluation Unit launched an online survey of Vanier scholar supervisors over the period of February to April 2014 that provided quantitative data to help address a range of evaluation issues and questions. As SSHRC and NSERC do not keep track of the identities and contact information for Vanier supervisors, SSHRC and NSERC Vanier scholars who were in their doctoral program at time of application and who listed a primary doctoral supervisor as a referee in their Vanier application were asked to confirm the identity of their supervisor as part of the survey of scholars. For NSERC and SSHRC Vanier recipient survey respondents whose supervisor information was incorrect or for Vanier scholars whose supervisor information as missing, individuals were asked to consent to receiving and forwarding a future email message from CIHR to their supervisor for the doctoral studies in which they received Vanier CGS support. The email contained a link to a one page questionnaire that enabled supervisors to provide CIHR with their name and contact information in order to receive invitations to participate in the survey.

A post-stratification weighting scheme was developed using a sampling weight to correct for disproportionate cases within each strata as compared to the population (i.e., federal granting agencies/area of research) (Table 4.4).

Table 4.3 – Vanier CGS Scholar Supervisor Survey Response Rate
Population Size Invitations Sent Completes Incompletes # Terminated Response Rate Margin of Error
(95% CL)
CIHR Vanier Supervisors 280 252 118 12 0 47% 6.9%
NSERC Vanier Supervisors 277* 88 69 3 0 78% 10.2%
SSHRC Vanier Supervisors 273* 84 48 7 1 57% 12.9%
Vanier Scholar Supervisors Total 830 424 235 22 1 55% 5.4%

Source: ResearchNet; Survey of Vanier Scholar Supervisors

* Although confirmed identities and contact info of supervisors of NSERC and SSHRC funded Vanier scholars were not available from administrative records, an assumption was held that every Vanier scholar has at least one primary doctoral supervisor whose field of research aligns to some degree to the field of research for the granting agency in which the application was reviewed.

Table 4.4 – Vanier Scholar Supervisor Survey Weighting Scheme
A. Population Size (N) B. Sample Size
(n)*		 C. Sampling Ratio (B/A) Proportional Weight
(A/830) / (B/258)
CIHR Vanier Supervisors 280 130 0.46 0.67
NSERC Vanier Supervisors 277 72 0.26 1.20
SSHRC Vanier Supervisors 273 56 0.21 1.52
Vanier CGS Supervisors Total 830 258 0.31 -

Source: ResearchNet; Survey of Vanier Scholar Supervisors

* includes complete and incomplete responses.

Appendices

Vanier CGS Matrix of Questions and Indicators

Evaluation Question Indicator Method
Performance - Achievement of Immediate Outcomes: Capacity Development
1. To what extent has the Vanier CGS program achieved its expected immediate outcomes?

1.1. Has the Vanier CGS program attracted and recruited world- class students?

1.1.1. Analysis of Vanier CGS nominee and recipient profiles, e.g.:

  • Language
  • Gender
  • Country of citizenship
  • Mobility
  • Institution
  • Department
  • Administrative data analysis

1.1.2. Perceptions of Vanier CGS recipient supervisors regarding:

  • Extent to which supervised Vanier CGS recipient(s) demonstrated academic, research, and leadership attributes and accomplishments
  • Comparison of demonstrated academic, research and leadership attributers and accomplishments of supervised Vanier CGS recipients with other doctoral students
  • Survey of Vanier CGS recipient supervisors
1.1.3. Assessment of quality of Vanier nominees (the extent to which students nominated for a Vanier CGS award were world-class)
  • Expert panel review
1.1.4. Perceptions of university representatives on the extent to which students applying for university nomination and students submitted for nomination by Canadian universities were world- class
  • Key informant interviews with university representatives

1.1.5. Number and type of promotional activities/venues to promote the Vanier program nationally and internationally

1.1.6. Extent of reach of Vanier program promotional activities/venues, e.g.:

  • Number of tradeshow attendees
  • Number of conference attendees
  • Number of online website visits
  • Administrative data analysis
  • DFATD Edu-Canada Marketing Initiative data
  • Key informant interviews with university representatives
1.1.7. Identified ways in which Vanier applicants became aware of the program
  • Survey of Vanier CGS recipients and non-recipients
1.2. Have Vanier CGS recipients received training that has led to their research, leadership and academic development?

1.2.1. Extent of Vanier CGS recipients and non-recipients involvement and satisfaction with supervisor and research environment:

  • Types and extent of skills (research, teaching, administration, mentorship and other professional development activities) acquired by Vanier recipients and non-recipients
  • Frequency and type of interactions of Vanier CGS recipients and non-recipients with supervisor and other faculty
  • Number and type of research contributions and knowledge dissemination activities undertaken by Vanier recipients and non-recipients
  • Type and number of research, teaching and service-related activities undertaken by Vanier recipients and non-recipients
  • Extent to which Vanier recipients and non-recipients are satisfied with their involvement in different stages of academic and non-academic career skills development
  • Type, number and value of grants, awards and prizes held by Vanier recipients and non-recipients
  • Number and value of research assistantships, teaching assistantships and other employment opportunities undertaken
  • Survey of Vanier CGS recipients and non-recipients
1.2.2. Perceptions of, and evidence provided by, university representatives of the impact of Vanier recipients on training/research environment at their university
  • Key informant interviews with university representatives
1.2.3. Perceptions of Vanier CGS recipient supervisors on the extent to which award recipients have had an impact on the training/research environment at their department
  • Survey of Vanier CGS recipient supervisors
1.2.4. Identified university leadership activities and opportunities provided to Vanier recipients
  • Key informant interviews with university representatives
1.2.5. Identified supervisor leadership activities and opportunities provided to Vanier recipients
  • Survey of Vanier CGS recipient supervisors
1.2.6. Extent to which Vanier recipients and non-recipients are satisfied with the opportunities available to develop their research, teaching, service and personal/professional skills
  • Survey of Vanier CGS recipients and non-recipients
1.3. Have Vanier CGS recipients received training that meets their career objectives?

1.3.1. Identified career objectives of Vanier CGS recipients and non-recipients, e.g.,:

  • Postdoctoral fellowship
  • Academia
  • Government
  • NGO
  • Private sector

1.3.2. Extent to which Vanier recipients and non-recipients indicate that their training prepared them for their career

1.3.3. Identified challenges and gaps in training received by Vanier recipients and non-recipients

1.3.4. % of Vanier recipients who obtained a Canada Graduate Scholarship - Michael Smith Foreign Study Supplement (CGS-MSFSS) to study abroad for part of their doctoral degree

1.3.5. Perceptions of Vanier and CGS-MSFSS recipients on the supplement and studying abroad
  • Survey of Vanier CGS recipients and non-recipients
Performance - Progress Towards Achievement of Intermediate Outcomes
2. To what extent has the Vanier CGS program progressed towards the achievement of its expected intermediate outcomes? 2.1.1. Perceptions of, and evidence provided by, university representatives regarding the extent to which the Vanier CGS program has enhanced the reputation of Canadian universities
  • Key informant interviews with university representatives
2.1.2. Perceptions of Vanier CGS recipient supervisors regarding the extent to which the program has enhanced the reputation of their university department
  • Survey of Vanier CGS recipient supervisors
2.1.3. Perceptions of, and evidence provided by, university representatives regarding the extent to which the Vanier CGS program has enhanced the capacity of their university to attract the best and brightest students
  • Key informant interviews with university representatives
2.1.4. % Vanier CGS recipients and non-recipients that have remained in Canada after completion of their studies

2.1.5. % of Vanier CGS recipients and non-recipients that indicate an intention to remain in Canada after the completion of their studies

2.1.6. Identified factors that influenced Vanier CGS recipients and non-recipients decisions to remain in Canada after completion of their studies

2.1.7. % of Vanier CGS recipients and non-recipients that obtained employment in fields/sectors relevant to their career objectives and doctoral training
  • Survey of Vanier CGS recipients and non-recipients
Performance – Progress Towards Achievement of Long-Term Outcomes
3. To what extent has the Vanier CGS program progressed towards the achievement of its expected long-term outcomes? 3.1.1. Extent to which Vanier recipients, non-recipients and recipient supervisors view the Vanier scholarship as prestigious in Canada and Internationally
  • Survey of Vanier CGS recipients and non-recipients
  • Survey of Vanier CGS recipient supervisors
Performance - Efficiency and Effectiveness: Program Design and Delivery
4. Has the Vanier CGS program been implemented as intendedFootnote 9?

4.1.1. Analysis of Vanier CGS university quota allocations, e.g.:

  • Distribution of allocations across universities
  • Proportion of allocations met, unmet and exceeded in total and across universities
  • Administrative data analysis
  • Document review
4.1.2. Identified reasons for any unmet Vanier CGS nomination quotas for Canadian universities

4.1.3. Identified review process used (including assessment criteria applied) by Canadian universities to select Vanier nominees

4.1.4. Perceptions on university review process used (including assessment criteria) including identified strengths, weaknesses and suggested improvements
  • Key informant interviews with university representatives
4.1.5. Perceptions on the Vanier CGS university quota system including identified strengths, weaknesses and potential improvements
  • Key informant interviews with Vanier CGS program staff and university representatives
4.1.6. Associations among Vanier CGS recipients and non-recipient profiles (e.g., gender, country of residency, mobility) and their federal granting agency selection committee ratings (overall and by selection criteria), rankings and funding decisions
  • Administrative data analysis
4.1.7. Extent to which Vanier CGS recipients and non-recipients express satisfaction with the application and selection process
  • Survey of Vanier CGS recipients and non-recipients

4.1.8. Comparison of Vanier CGS program design and implementation with similar national and international scholarship in terms of:

  • Program objectives
  • Award value and duration
  • Selection process
  • Funding competition statistics

4.1.9. Identified best practices among national and international scholarship programs
  • Organizational scan
4.1.10. Identified strengths of, and suggested improvements to, the design and implementation of the Vanier CGS program

4.1.11. Identified facilitators and inhibitors to attracting and recruiting foreign world-class doctoral students

4.1.12. Perceptions of comparable or alternative delivery mechanisms to the Vanier CGS program
  • Key informant interviews with Vanier CGS program staff and university representatives
  • Survey of Vanier CGS recipients and non-recipients
  • Survey of Vanier CGS recipient supervisors
4.1.13. Identified mechanisms to support Vanier CGS recipients in networking with other scholars and receiving program-related updates and communications
  • Survey of Vanier CGS recipients
5. Has the Vanier CGS program been delivered by the federal granting agencies in a cost efficient manner? 5.1.1. Ratio of operating expenditures to applications reviewed in total and by federal granting agency

5.1.2. Ratio of operating expenditures to grants funds awarded in total and by federal granting agency

5.1.3. Comparison of existing ratios of operating expenditures to applications reviewed/grant funds awarded with federal granting agency programs (benchmarking)

5.1.4. Proportion of Vanier CGS budget expended in total and per agency
  • Administrative data analysis
Relevance (Continued need; Alignment with federal government priorities, roles and responsibilities)
6. Is there a continued need for the Vanier CGS program?

6.1.1. Identified need for support for world-class doctoral students in Canadian universities:

  • Assessment of supply and demand for Canadian highly qualified personnel (HQP)
  • % of Vanier CGS non-recipients that enrolled in doctoral studies outside of Canada
  • % of Vanier CGS non-recipients that enrolled in doctoral studies in Canada
  • % of Vanier CGS recipients who indicated they would have pursued doctoral studies outside of Canada without Vanier funding
  • Proportion of Vanier CGS university applications for nominations received versus number of nominations submitted in total and across universities
  • Proportion of allocations met, unmet and exceeded in total and across universities
  • % of Vanier CGS recipients who decline award and identified reason(s) why
  • % of Vanier CGS recipients and non-recipients who indicated concern with their financial situation prior to application to their doctoral program
  • Administrative data
  • Document review
  • Survey of Vanier CGS recipients and non-recipients
6.1.2. Extent of duplication/overlap of Vanier CGS program objectives with other federal granting agency doctoral student funding programs
  • Organization scan
7. Does the Vanier CGS program align with government priorities and is it consistent with federal roles and responsibilities? 7.1.1. Compatibility of Vanier CGS program objectives/expected results with federal granting agencies’ and federal government’s priorities

7.1.2. Identified role for the federal government in supporting world- class doctoral students in Canada
  • Document review
  • Key informant interviews with Vanier CGS program staff and university representatives and agency/ government officials

Core Treasury Board Evaluation Issues

Relevance
Issue #1: Continued Need for program Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians
Issue #2: Alignment with Government Priorities Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes
Issue #3: Alignment with Federal Roles and Responsibilities Assessment of the role and responsibilities for the federal government in delivering the program
Performance (effectiveness, efficiency and economy)
Issue #4: Achievement of Expected Outcomes Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes
Issue #5: Demonstration of Efficiency and Economy Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes

Cross-walk of Vanier CGS Evaluation Questions and TBS Core Issues

Vanier CGS Evaluation Question Core Treasury Board Evaluation Issue Addressed
1. To what extent has the Vanier CGS program achieved its expected immediate outcomes?

1.1. Has the Vanier CGS program attracted and recruited world-class students?

1.2. Have Vanier CGS recipients received training that has led to their research, leadership and academic development?

1.3. Have Vanier CGS recipients received training that meets their research and career objectives?		 Issue #4: Achievement of Expected Outcomes
2. To what extent has the Vanier CGS program progressed towards the achievement of its expected intermediate outcomes? Issue #4: Achievement of Expected Outcomes
3. To what extent has the Vanier CGS program progressed towards the achievement of its expected long-term outcomes? Issue #4: Achievement of Expected Outcomes
4. Has the Vanier CGS program been implemented as intended? Issue #4: Achievement of Expected Outcomes
5. Has the Vanier CGS program been delivered by the federal granting agencies in a cost efficient manner? Issue #5: Demonstration of Efficiency and Economy
6. Is there a continued need for the Vanier CGS program? Issue #1: Continued Need for program
7. Does the Vanier CGS program align with government priorities and is it consistent with federal roles and responsibilities? Issue #2: Alignment with Government Priorities

Issue #3: Alignment with Federal Roles and Responsibilities

References

CIHR. (2009). Health Research Roadmap: Creating innovative research for better health and health care. CIHR’s Strategic Plan 2009-10 - 2013-14.

Department of Finance Canada. (2008). The Road to Balance: Creating Jobs and Opportunities [ PDF (3 MB) - external link ].

Department of Finance Canada. (2008). The Budget Plan 2008: Responsible Leadership [ PDF (1.54 MB) - external link ].

Employment and Social Development Canada. (2011). Canadian Occupational Projection System (COPS): University Professors and Assistants.

NSERC. (2014). Report on Plans and Priorities.

Science, Technology and Innovation Council. (2011). State of the nation 2010: Canada’s science, technology and innovation system.

SSHRC. (2013) Strengthening Canada’s Cultures of Innovation: Strategic Plan 2013-16 [ PDF (261.79 KB) - external link ].

Statistics Canada (2014). Education Indicators in Canada: An International Perspective: 2013 [ PDF (4.99 MB) - external link ]. (Catalogue no. 81-604-X).

Statistics Canada. (2011). Expectations and Labour Market Outcomes of Doctoral Graduates from Canadian Universities [ PDF (340.98 KB) - external link ]. (Catalogue number 81-595-M No. 089).

Statistics Canada. (2008). Doctoral Education in Canada: Findings from the Survey of Earned Doctorates 2005/2006. (Catalogue number 81-595-M — No. 069).

Tamburri, R. (2013). The PhD is in need of revision. University Affairs.

Footnotes

Footnote 1

Canadian citizens and permanent resident holders of Vanier scholarships may also apply for a CGS Michael Smith Foreign Study Supplement of up to $6,000 to help offset the costs of undertaking research studies outside Canada for a period of three to six months of their degree.

1

Footnote 2

The Vanier program was harmonized in 2009 which involved adopting a uniform application and selection process across the three federal granting agencies and involved the allocation of nomination quotas across eligible Canadian universities.

2

Footnote 3

Eligible universities receive three-year allocations based on 600 total nominations per research granting agency (CIHR, NSERC and SSHRC). The distribution of allocations among eligible universities is based on the following method: for Canadian universities that have a doctoral-level program in the subject areas supported by one or more of the granting agencies, the calculation is based on the sum of the three-year rolling funding average used for the last Canada Research Chairs calculations (2007-08, 2008-09 and 2009-10), and the payments made under the respective granting agencies for the Canada Graduate Scholarships doctoral awards for the three most recent fiscal years (2008-09, 2009-10, and 2010-11). Universities have the flexibility to submit as many applications as they wish in any given year, within their maximum three-year allocation per granting agency. Universities cannot exceed their three-year allocations or transfer allocations between agencies.

3

Footnote 4

See Triennial Nomination Allocations by University for 2013, 2014 and 2015 Vanier Canada Graduate Studies Scholarships Competitions [ PDF (91.65 KB) - external link ] for allocations per university.

4

Footnote 5

For the purpose of the evaluation, foreign Vanier CGS scholars were defined as residing outside Canada and being neither Canadian citizens or permanent residents of Canada at the time of their application.

5

Footnote 6

Vanier scholar supervisors who indicated they supervised more than one award holder were asked to respond throughout the survey to their first award holder.

6

Footnote 7

Vanier scholar supervisors rated leadership on a nine point scale with ‘exceptional’ being the highest positive rating.

7

Footnote 8

The role of the selection board to oversee processes, policies and results to ensure that the program was meeting its objectives was transitioned to other existing governance bodies. Membership of the board included a distinguished Canadian as Chair, a member of the Diplomatic Corps, the Chairs of the Boards of each Agency, world renowned researchers and the Deputy Minister of Industry Canada as an observer.

8

Footnote 9

The evaluation will assess whether Vanier CGS has been implemented as intended over the lifecycle of the program, taking into account various changes made over time to design and/or delivery.

9

Date modified: