Evaluation of the Canada Graduate Scholarships (CGS) Program, 2008-2013

Final Report: September 2016

Acknowledgements

Most of the data collection and analyses (online surveys, focus groups and key informant interviews) for this evaluation were conducted by Goss Gilroy Inc., Management Consultants, who were also responsible for the initial drafts of the final report and supporting technical reports. We specifically acknowledge the contributions of Celine Pinsent, Pierre Mercier and Sandy Moir.

The CIHR Evaluation Unit implemented the document review, administrative data analysis and key informant interviews with senior CGS program management. The draft report was finalized by the CIHR Evaluation Team with input from the Evaluation Working Group (EWG).

Thanks to the original members of the EWG that started this evaluation project: Robert McLean, Patrick MacGuire, Heather Bell, Michael Goodyer, Martin Rubenstein, Anna Engman, Shannon Clark-Larkin, Susan Morris, Delphine Bouilly, Gordana Krcevinac, Annik Poirier, Peggy Borbey, Craig McNaughton, and Serge Villemure.

Current members of the EWG are: Kwadwo (Nana) Bosompra, Sarah Viehbeck, Michael Goodyer, Julie Conrad, Frank Zegers, Tiffany Richards, Shannon Clark-Larkin, Kenneth Stephenson, and Alison Surette.

Special thanks to all the evaluation participants – survey respondents, focus groups and interview participants, CGS program management, the Interagency Evaluation Steering Committee and the Tri-agency CGS Harmonization Steering Committee.

The CIHR Canada Graduate Scholarship Evaluation Team:
Kwadwo (Nana) Bosompra, PhD; Sarah Viehbeck, PhD; Michael Goodyer, MPA; David Peckham, MSc.

Table of Contents

• List of Acronyms
• Executive Summary
  • Context
  • Program Description
  • Evaluation Purpose, Scope and Methodology
  • Summary of findings
  • Recommendations
• 1.0 Introduction
  • 1.1 Program Profile
  • 1.2 Objectives and Scope of the Evaluation
  • 1.3 Methodology
• 2.0 Performance
  • 2.1 Immediate Outcomes
  • 2.2 Intermediate Outcomes
  • 2.3 Exploration of Long-term Outcomes
  • 2.4 Economy and Efficiency
• 3.0 Relevance
  • 3.1 Continued need for the CGS program
  • 3.2 Consistency of CGS program with federal roles and responsibilities
  • 3.3 Alignment of CGS program with Government of Canada priorities
• 4.0 Conclusions and recommendations
  • 4.1 Performance
  • 4.2 Economy and efficiency
  • 4.3 Relevance
  • 4.4 Recommendations
• Appendix A: CGS Program Logic Model
• Appendix B: Methodological Details
  • B.1 Key Informant Interviews
  • B.2 Focus Groups
  • B.3 Document Review
  • B.4 Surveys
• Appendix C: Outcomes by Degree Level and Study Area

List of Acronyms

Executive Summary

Context

This report presents the results of the second evaluation of the Canada Graduate Scholarships (CGS) program and covers fiscal years 2008-09 to 2012-13.

Since the introduction of the CGS in 2003 and its first evaluation in 2008, the graduate level training landscape in Canada has undergone substantial change, including the introduction of new awards (e.g., Vanier and Banting).

It is in this evolving context, CGS design changes and implementation of the results of completed, ongoing and planned evaluations of tri-agency and agency-specific training programs that this second evaluation of the CGS must be viewed. The evaluation focuses on program performance and relevance with comparisons, to the first CGS evaluation as well as agency-specific (NSERC and SSHRC scholarships) and tri-agency (Vanier CGS Doctoral) training program evaluations.

Program Description

The CGS program was launched by the federal government in 2003 with the broad objective of ensuring a reliable supply of highly qualified personnel (HQP) to meet the needs of Canada's knowledge economy. The program provides financial support to Canadians and permanent residents pursuing master’s or doctoral studies in Canadian universities and has the following expected outcomes:

• Increase the incentives for students to enroll in graduate studies in Canada;
• Increase the enrollment in graduate studies in Canada;
• Increase the incentives for scholarship recipients to complete studies within a specific time period;
• Increase the recognition by the research community of the federal government’s financial support for research training;
• Increase the numbers of students completing degrees and doing so in a timely manner;
• Increase high-quality research training as well as the ability to attract and retain experienced researchers;
• Increase the capacity to meet the demand for highly qualified personnel in Canadian universities and in the public and private sectors; and,
• Improve the branding of Canada as a home of research excellence and Canadian universities as world-class research centres.

The CGS master’s (CGS-M) and CGS doctoral (CGS-D) levels are each allocated about 2,500 awards annually and are distributed across the tri-agencies. The CGS-M is $17,500 for one year while the CGS-D is worth $35,000 annually for up to three years.

In 2013, program delivery processes for the CGS-M were harmonized to improve efficiency and work to harmonize the CGS-D is in progress.

Evaluation Purpose, Scope and Methodology

The evaluation covers fiscal years 2008-09 to 2012-13 and assesses the outcomes of successful CGS applicants (referred to throughout this report as CGS recipients) in comparison with unsuccessful CGS applicants who also did not receive a Vanier scholarship or any other Agency-specific award (applicants). The evaluation assesses the longer-term impact of the CGS program by analyzing the extent to which the program is contributing to the supply of HQP required by Canadian universities, and public and private sectors in the knowledge economy.

Multiple lines of evidence were used to triangulate the evaluation findings including a document review, administrative data, key informant interviews, focus groups, and surveys of recipients and applicants.

Summary of findings

Performance

To what extent has the CGS achieved its expected immediate outcomes?

The extent to which the CGS program is achieving its immediate outcomes is mixed. In relation to its intended role as an incentive for enrolment in graduate studies, the evaluation finds that the self-reported primary motivation for students to pursue a graduate degree remains a deep interest in the area of study. This is consistent with the finding that the majority of the students (84.5%) were already enrolled before obtaining their CGS or state that they would have enrolled regardless; whereas only about one in ten (13.2%) would not have enrolled in a program had they not received a CGS award.

Graduate enrolment in Canada ranged from ~160,000 in 2007-08 to ~190,000 in 2012-13. The CGS target of funding 5,000 scholarships a year cannot, on its own, increase enrollment numbers significantly. That said, the CGS has made a clear contribution to the ability of students to devote more time to their studies. CGS recipients abandon their studies less frequently and accumulate less debt than applicants. Nevertheless, the total duration of their studies remains strongly affected by concurrent factors such as the nature of the research process itself.

The CGS is well-known within academia and is seen as a part of the suite of federal training awards.

To what extent has the CGS achieved its expected intermediate outcomes?

The CGS is achieving several of its intermediate outcomes. Students and administrators view the scholarship as a means to earn a living while studying full-time. A larger proportion of CGS-D recipients (64.4%) than applicants (59.9%) completed their degree within the time frame of the survey and, among those who have completed their degree, recipients completed slightly faster than applicants (60 months versus 64 months). Recipients attribute the timely completion of their studies at least in part to the CGS providing support for living expenses; applicants cite the converse.

The amount of the CGS award is seen as affording opportunities that enrich the research training experience for recipients (such as conference attendance). Survey data confirms that recipients are more involved in research-related activities and are more productive in terms of publications and communications than applicants. The evaluation found that CGS recipients and applicants were satisfied with opportunities to develop their research skills and personal/professional skills, with recipients more satisfied than applicants. In addition, CGS is perceived as helping attract highly qualified researchers to universities which creates a synergy whereby universities with highly qualified researchers are, in turn, attractive to top graduate students, including CGS recipients.

Graduates find employment related to their studies, with CGS recipients performing better than applicants. Recipients also reported that personal/professional experience and research-related experience helped them in obtaining the position they currently hold. A majority of doctoral graduates are largely employed in university settings whereas the majority of master’s are employed in the private sector or government. This evidence indicates that the CGS program has contributed to increasing the capacity to meet demand for HQP in the faculties of Canadian universities and in the public and private sectors.

The CGS is recognized as distinctive by professors, graduate students and administrators in Canada and promotes Canadian research excellence abroad through their support for increased productivity in terms of publications and communications worldwide. The evaluation found that the multiplicity of scholarship names within the CGS program may hinder brand recognition.

What outcomes have been achieved by CGS recipients following their direct experience with the program?

The CGS program has contributed to long-term impacts on the career paths of participants; generally validating recipients’ decision to pursue a research career, facilitating the time spent studying and perfecting research and other skills. This is demonstrated by the findings that CGS recipients produce a greater number of articles and presentations, are more likely to be currently employed in a position closely related to their degree program and earn more than applicants.

Overall, CGS recipients have slightly more international exposure than applicants thus extending Canada's reputation abroad. While CGS recipients moderately valued opportunities to gain some international experience, overall they did not value the opportunity to pursue a full graduate degree at an institution outside of Canada and the same was true of applicants. In addition, recipients had little interest in completing their degree abroad if the CGS scholarship had allowed it. It should be noted that since CGS awards cannot be held outside Canada, students who want to pursue international studies would typically opt for an agency-specific scholarship which may explain the low levels of interest observed in this evaluation. CGS recipients can obtain some international experience through the CGS Michael Smith Foreign Study Supplement (MSFSS) which is a one-time award of up to $6,000 to undertake 3-6 months of study outside Canada. Participation in the CGS-MFSS is restricted to 250 awards per year and recipients are quite satisfied with the experience.

Economy and Efficiency

Available evidence suggests that the tri-agencies are delivering the CGS program in a cost-efficient manner. For the period from 2009-10 to 2013-14, administrative expenditure (direct attributable costs only) as a proportion of total expenditure (direct administrative costs and award expenditure) ranged from 1.7% to 1.9%. The average for the period is 1.8% as compared to 3.5% for the Vanier CGS program. The difference in ratios could be partially due to the effects of economies of scale given that there is a fixed cost to setting up the basic administrative structures for running any program. The expenditures presented in the analyses do not include indirect and direct non-attributable costs which could be substantial. Therefore, the expenditures are an underestimation of the total costs associated with the program and are computed this way to allow for comparability with the 2014 Vanier CGS evaluation results.

Recipients appreciate the value of the scholarships but question its duration in relation to actual time to degree completion. Both applicants and recipients perceive the adjudication process as opaque and would like more information about how applications are scored and more feedback on applications. The CGS-M harmonization process has improved deadlines, the application process, and the distribution among agencies according to areas of expertise. There were some reservations about the availability and clarity of the information from the three agencies, about the full readiness of the supporting technology, and payment processes.

Relevance

Does the CGS program remain relevant?

The evaluation findings indicate the continued need for the CGS program to foster excellence in graduate studies and research through financial support, enabling high achieving students to pursue graduate degrees regardless of their financial means and to devote more time to their studies, thus being more productive. There is however, mixed evidence for the extent to which the program is meeting some of its immediate objectives (e.g., CGS as an incentive to enroll in graduate studies) in that it cannot be demonstrated that the objectives are being met. The CGS program aligns with federal roles and responsibilities and the mandates of the tri-agencies to develop HQP who can contribute to the growth of Canada’s knowledge economy as outlined in the 2014 Science, Technology and Innovation Strategy. Additionally, the program is consistent with federal government and tri-agency priorities.

Recommendations

The evaluation evidence indicates that the CGS program is effective, continues to be relevant, and is needed to support HQP development to insure that Canada’s knowledge economy remains globally competitive well into the future. While evidence to support the achievement of some of its immediate outcomes is mixed, the program is achieving several of its intermediate outcomes. The available evidence indicates that the program is being run efficiently. The following recommendations are made:

1. Review and revise the Canada Graduate Scholarship program’s expected outcomes and strengthen performance measurement.

Since the introduction of the CGS in 2003 and the 2008 evaluation, the graduate level training landscape in Canada has undergone substantial change. Newer programs - the Vanier CGS and Banting postdoctoral fellowship programs - have been introduced to attract and retain the best doctoral and postdoctoral level trainees respectively and the objectives of these programs have created, de facto, a new hierarchy across the suite of federally-funded training awards.

The evaluation found evidence of the program’s relevance and the broad need for the CGS, though there is mixed evidence for the extent to which the program is meeting its specific objectives. In particular, the evaluation found that the CGS program has limited ability to increase incentives for, or enrollment in, graduate studies. This finding calls into question the logic underlying the program as it cannot be demonstrated that two of the four immediate outcomes of the program are being achieved; namely, the outcomes to increase incentives to enrol and increase enrollment. This finding is consistent with the 2008 evaluation findings and recommendation to rethink the program logic.

The evaluation could have benefited from good quality performance data. A performance measurement strategy and end of award reporting tool should be developed and implemented to enable ongoing tracking of recipients.

2. In the context of Canada Graduate Scholarship harmonization across the Tri-Agencies, the program should provide more information on the review process and outcomes to applicants, and also explore opportunities for branding the program under a single name.

Plans for harmonizing the CGS-D are proceeding and preliminary evaluation findings have informed aspects of the process including validating the newly proposed core principles for the CGS-D and clarifying program objectives. In the context of harmonization and the increased role for institutions in a harmonized program, the CGS program should work to improve transparency of review processes and feedback to applicants. Although the evaluation found, perhaps not surprisingly, that recipients were more satisfied with the fairness of the selection process than applicants, both groups found the selection process unclear. This suggests that more information on the application process and more feedback to applicants would be valuable, as would prompt notification of the outcome of their application.

The evaluation found that the CGS program is recognized as distinctive; however, the multiplicity of scholarship names within the program may hinder brand recognition.

1.0 Introduction

This report presents the results of the second evaluation of the Canada Graduate Scholarships (CGS) program and covers fiscal years 2008-09 to 2012-13. The purpose of the evaluation is to provide insightful and valid findings about the performance and relevance of CGS for Tri-Agency management in accordance with the Treasury Board Secretariat’s Policy on Evaluation. The evaluation focuses on program performance and relevance with comparisons where feasible, to the first CGS evaluation (2008) as well as the recently completed agency-specific (NSERC and SSHRC scholarships) and Tri-Agency (Vanier CGS Doctoral) training program evaluations. The evaluation was led by the Canadian Institutes of Health Research (CIHR) in collaboration with the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC).

1.1 Program Profile

1.1.1 Program Description

Established by the Government of Canada in 2003 and awarded through CIHR, SSHRC and NSERC, the CGS provides financial support to Canadians and permanent residents pursuing master's (CGS-M) or doctoral studies (CGS-D) in Canadian universities. Additional funding is also available to CGS award holders through the Michael Smith Foreign Study Supplement (CGS-MSFSS) to study abroad for up to six months of their degree.

The program’s objectives were established at its inception in 2003 and documented in the program Terms and Conditions and Performance Measurement Framework (see Appendix A for the Logic Model which illustrates the CGS program’s intended outcomes). The overall objective of the program is to ensure a reliable supply of highly qualified personnel (HQP) to meet the needs of Canada's knowledge economy. The CGS is intended to increase incentives for graduate studies in Canada and to brand the scholarship as a prestigious award that is internationally competitive (CGS Terms and Conditions 2009). The more specific objectives of the CGS program are presented in Figure 1.

Figure 1: CGS Program Overview

Source: CGS Terms and Conditions 2009 and CGS Program Logic Model.

CGS awards are primarily a stipend, but may also include a research allowance component of approximately 15% of the value of the award. An individual can only hold one CGS-M and one CGS-D award in their lifetime, the latter holding a duration of up to three years. NSERC and SSHRC have been granted special permission to place the unused balance of awards in a given fiscal year in a General Graduate Studies Fund, one for each granting agency, to be redirected to other scholarship activities. Program funding is allocated to each Agency according to the estimated disciplinary distribution of the graduate student community: SSHRC – 52%, NSERC – 32% and CIHR – 16%. The proportions for the MSFSS are slightly different: SSHRC (50%), NSERC (32%) and CIHR (18%). Table 1 shows the number of CGS recipients per fiscal year.

Table 1 - Number of CGS Awards Being Funded by Agency^{Footnote 1}, 2009-10 to 2012-13

Due to the higher proportion of CGS awards allocated to SSHRC, the agency has the highest annual appropriations per fiscal year while CIHR has the lowest (Figure 2).

Figure 2 - Agency Appropriations for the CGS Program from 2009-10 to 2012-13

Source: Tri-Agency data.

1.1.2 Program Delivery

The criteria used for selecting candidates to the program at the doctoral level are not yet harmonized across the Tri-Agencies. Currently, each Agency applies its own application and selection process for CGS-D awards and each has flexibility in determining what other criteria should be used in reviewing scholarship applications. Therefore, the excellence of a candidate may be assessed differently within each Agency. For instance, SSHRC and NSERC place greater weight on academic performance and CIHR places more weight on the characteristics and abilities of the candidate. Canadian universities have an important role as co-deliverers of the CGS Program. In the current CGS-D delivery model, universities preselect candidates for the NSERC and SSHRC CGS-D awards while CIHR CGS-D candidates apply directly to CIHR’s doctoral award program. Many of these varying features will be streamlined into a single program delivery structure when the CGS-D harmonization is fully implemented.

The process is different at the master’s level since the CGS-M harmonization has already been implemented. Program delivery of the CGS-M has been devolved to the academic institutions which have allocations. Students apply to the academic institution where they would like to hold the CGS-M and award recipients are selected by the academic institution.

At both the master’s and doctoral levels, CGS awards are not paid directly to students; instead they are paid to recipient institutions who administer the funds (both stipends and where appropriate research allowances) on each granting agency’s behalf.

1.1.3 CGS Harmonization

Due to limitations in operational funds, when the CGS was announced by the federal government in 2003, the Tri-Agencies implemented the program using their respective existing business models and delivery mechanisms for Agency-specific awards (Project Charter, CGS-D Harmonization Project). Although significant savings were made, the delivery mechanisms were disparate across the three agencies and over time these differences were found to be inhibiting program delivery efficiency. Additionally, in response to the changing needs of Canadian and international graduate students, to demonstrate a commitment by the Agencies to work together, and to eliminate the silos and duplication that existed in key program areas, the presidents of the tri-agencies, in the summer of 2012, committed to harmonizing their processes by forming a Tri-Agency Harmonization Team to redesign the CGS Program delivery process.

Guided by five core principles: integration, simplicity, quality of service, excellence, and accountability, the harmonization project was expected to streamline application and adjudication procedures and processes and to be fully implemented by 2018. The master’s component of the CGS (CGS-M) program was harmonized in fall 2013 with a single application portal for all students and common application forms, program policies, such as eligibility, and post award practices and plans for harmonizing the doctoral component are proceeding.

The program’s terms and conditions were renewed in August, 2014 and the tri-agencies are planning a revision to align with any design changes arising out of their current planning processes. At this same time, the tri-agencies have been evaluating their agency-specific training programs along with tri-agency programs like the Vanier Doctoral Award Program and the Banting Postdoctoral Fellowships Program. It is in this evolving context of CGS design changes and implementation of the results of completed, ongoing and planned evaluations of tri-agency and agency-specific training programs that this second evaluation of the CGS must be viewed.

1.2 Objectives and Scope of the Evaluation

The evaluation will determine whether or not the program has achieved its stated objectives (from current Terms and Conditions of the program), what unexpected outcomes it may be creating or contributing to, how efficiently and effectively program delivery is occurring, and how relevant the CGS Program is. The evaluation questions were developed in consultation with Agency Evaluators, Agency CGS Directors, the CGS Harmonization Team and an external CGS recipient. The evaluation questions are presented by TBS 2009 Directive on Evaluation core issue in Table 2.

This second evaluation of the CGS program primarily covers fiscal years 2008-09 to 2012-13 and includes some data analyses for fiscal years 2003-04 to 2007-08 to enable direct comparisons with the 2008 CGS evaluation. Broadly speaking, the evaluation assesses the outcomes of successful CGS applicants (referred to throughout this report as CGS recipients) in comparison with CGS applicants who also did not receive a Vanier scholarship or any other Agency-specific award (referred to in this report as applicants). The evaluation also assesses the longer-term impact of the CGS program through the conduct of a trajectory analysis, in particular, the extent to which the program is contributing to the supply of HQP required by Canadian universities, and public and private sectors in the knowledge economy. Both CGS recipients and applicants were queried on issues such as: their current country of residence, employment, career stage, the number of research grants and awards applied for and received (particularly through the federal granting agencies), their research productivity and the perceived alignment of their CGS award(s) and education with their current and expected future employment.

To help determine the suitability of the current CGS program design and inform any decisions regarding changes to the program theory (i.e., links between program objectives and expected outcomes as currently stated in the logic model), the evaluation examines possible unexpected outcomes of the program through the use of focus groups with current CGS recipients and applicants as well as key informant interviews with University administrators, CGS recipients’ supervisors and private and/or public sector employers of CGS award holders.^{Footnote 2}

To determine if there have been changes in the experiences of CGS award holders over time, the views and experiences of CGS recipients are compared across two cohorts of competitions launched 2002-2007 and 2008-2011 via survey and via the 2008 evaluation report.

In addition to the comparisons over time and across groups of recipients and applicants, the analyses also provide a breakdown of results by self-reported domain or area of study^{Footnote 3} (Social Sciences and Humanities (SSH), Natural Sciences and Engineering (NSE), and Health Sciences (HS) in order to better understand relative successes and limitations of the CGS program in each context.

Table 2: Evaluation Questions

1.3 Methodology

Consistent with TBS guidance and recognized best practice in evaluation (e.g., McDavid & Hawthorn 2006), a range of methods was used to triangulate evaluation findings. The approach of using multiple methods involving both quantitative and qualitative evidence is designed to ensure that the evaluation findings are robust and credible and that valid conclusions can be drawn about the performance and relevance of the Program. Figure 3 summarizes the methods employed in the evaluation.

Figure 3: Methods Employed in CGS Evaluation

Survey response rates were 26% and 22% respectively for the 2002-2007 and 2008-2011 cohort of recipients and 11% and 13% respectively for the corresponding cohort of applicants and these low rates should be borne in mind in interpreting the results.

All statistical analyses were performed using IBM’s Statistical package for the Social Sciences (SPSS). The z-test (at an alpha level of 0.05 for the Type I error rate) including a Bonferroni adjustment where necessary, was used to assess the statistical significance of percentage differences while analyses of covariance (ANCOVA) was used for ratio level measures.

Survey respondents were free to skip any questions and any such non-response items were coded as missing and excluded from the inferential statistics. Therefore the sample sizes used for statistical tests are not constant throughout the report and for ease of comprehension the maximum sample sizes for the groupings used in tests of statistical significance are presented in Table 3. Notably, there were far fewer CGS recipients and applicants in the HS area of study than in the two other areas, and there were fewer applicants who answered the survey at the master's level. However, the use of ANCOVA eliminates the correlation effect due to unequal sample sizes^{Footnote 4}. In addition, in spite of the smaller sample size, the Standard Error of the Mean (SEM) was not much larger in HS than in the other areas.

Table 3: Maximum Sample Sizes for Groupings Used in Tests of Statistical Significance

The number of participants in the key informant interviews was small overall and within each key informant group, which made it difficult to form overall findings and draw conclusions. In particular, only three individuals were interviewed for the employer category and therefore these findings have not been included in the report.

Regarding focus groups, while the intent was to identify a random sample at each location, the contact information was not current for many of the individuals in the sample and therefore it was not clear how representative the participants were of CGS recipients and applicants. Those who have completed their studies or moved elsewhere are likely under-represented.

It should be noted that the expenditures discussed in the cost efficiency section do not include indirect and direct non-attributable costs which could be substantial. Therefore, the expenditures are an underestimation of the total costs associated with the program and are computed this way to allow for comparability with the 2014 Vanier CGS evaluation results.

Further details of the methodology including survey response rates and implications of differences in survey sample sizes by degree level and area of study are presented in Appendix B.

2.0 Performance

2.1 Immediate Outcomes

The program's immediate outcomes are to:

• Increase the incentive for students to enrol in graduate studies in Canada
• Increase the enrolment in graduate studies in Canada
• Increase the incentive for scholarship recipients to complete studies within a specific time period
• Increase the recognition by the research community of the federal government’s financial support for research training

2.1.1 Increase incentive for students to enrol in graduate studies in Canada

It is beneficial to have some understanding of what motivated CGS recipients and applicants to pursue graduate studies, prior to discussing the impacts that the CGS may have had on their decision. In the focus groups, the most frequently cited motivator for pursuing a master's or doctoral degree was intellectual curiosity and a passion for research. In some other cases, the primary motivation was a requirement to hold a PhD in order to work in their field (e.g. as a licensed psychologist). Similarly, other participants said that they were primarily motivated by a belief that having a master's or doctoral degree would distinguish them in the labour market. Some doctoral level applicants in SSH who had obtained a scholarship from the Quebec government noted that a significant scholarship although not necessarily the CGS, was needed in order to pursue a doctoral degree and stay in the program. The motivations for pursuing graduate studies appeared to be the same for CGS recipients and non-recipients.

The participants’ comments were echoed by almost all interviewed administrators and supervisors who felt that the CGS award on its own was not an incentive to enrol in graduate studies. These informants considered that students are already motivated to enter graduate studies based on interests or choosing careers that need a graduate degree such as in the health field. However, the CGS was considered to contribute to likelihood of pursuing a career in research because the award enables the students to focus on research as opposed to working part-time while enrolled in a graduate program.

The survey of scholarship recipients strongly corroborates the qualitative information When asked to rate the importance (on a 7-point scale) of different factors in their decision to enroll in a graduate studies program, CGS award holders indicated that their deep interest in the area of study, self-betterment or improvement, contributing to the improvement of humanity and the challenge of pursuing graduate studies for its own sake were generally more important than other factors and there were differences by study area (Figure 4). Securing another form of financial support (e.g. as noted in the focus groups by the SSH doctoral applicants who had obtained a Quebec government scholarship) and factors directly related to the CGS such as the prestige of the CGS, the possibility of receiving the award or actually receiving it were scored less, as somewhat important.

Figure 4: Average Importance of Various Factors in Decision to Enroll in Graduate Studies - CGS Recipients^{Footnote 5}

Source: Survey of CGS Recipients.

Other analyses not presented in Figure 4 showed that the order of importance, overall, was similar for master’s and doctoral students except for deep interest in area of study (average of 6.3 for doctoral vs. 6.1 for master’s), contribution to improvement of humanity (5.4 vs. 5.2) and necessary credentials for a desired position (5.1 vs. 4.7) where doctoral students tended to assign more importance than master’s students.

The moderate level of motivational importance given to the scholarship parallels the 2008 evaluation results where 70% to 75% of respondents across groups and degrees identified the scholarship as an important factor in their decision to enroll and 45% to 49% of respondents identified the prestige as an important factor in their decision to enroll. This contrasted with the much higher importance of the students’ deep interest in the area of study reported by 90% to 92% of recipients. Parallel to the perceptions of CGS recipients, applicants in the focus groups said that the CGS had not been a factor in their decision to pursue graduate studies. Some described the CGS as a tool that would facilitate their education but that they were prepared to fund their schooling through other means (e.g. working part-time, other awards, loans, and family support).

Even lower proportions were reported in the evaluation of SSHRC’s doctoral fellowships program in 2015 where 47% of recipients of the doctoral award vs. 61% of SSHRC’s CGS recipients in the 2008-2011 cohort of recipients considered the possibility of receiving the award to be moderately to extremely important.^{Footnote 6} In the NSERC scholarships and fellowships evaluation conducted in 2015, 61% of Industrial Postgraduate Scholarships/Industrial Innovation Scholarships (IPS/IIS), 51% of CGS, 42% of Postgraduate Scholarships (PGS) recipients and 47% of applicants considered it to be moderately to extremely important.^{Footnote 7}

As illustrated in Table 4, there is a drive to continue to pursue graduate school even without a scholarship. Nearly 85% of CGS recipients indicated that they were already enrolled (39.6%) when they won the award or would have still enrolled (44.9%) had they not won it. This share is even greater for recipients in HS, where over 90% of respondents would have pursued their plans or continued their enrolment (43.4% and 50.5% respectively). These percentages of continued enrollment even in the absence of a scholarship are higher than in the 2008 evaluation (65% for master's to 56% for doctorates)^{Footnote 8}. The proportion is similar to that reported in the SSHRC doctoral fellowships program evaluation where 84% of fellows were already enrolled (58%) or would still have enrolled (26%) had they not won it.^{Footnote 9} Similarly, the NSERC evaluation reported that 78% of PGS recipients and 64% of IPS/IIS recipients were already enrolled or would have enrolled in their doctoral program.^{Footnote 10}

Table 4: Path Most Likely Chosen Regarding Graduate Studies, Had They Not Received a CGS Award, CGS Recipients

In summary, after the main motivation of a deep interest in the field and several other factors have been taken into consideration (Figure 4) the possibility of receiving the CGS is considered a somewhat important factor in enrolling in graduate studies.

2.1.2 Increase enrolment in graduate studies in Canada

There were no direct comments from the administrators (in the key informant interviews) or the students (in the focus groups) about the actual increase in enrolment in graduate studies in Canada during the 2008-2012 period. Statistical data do show an increase (Figure 5)^{Footnote 12} but this trend is confounded in part with a major funding effort from the 2005 Ontario Budget’s Reaching Higher Plan committed to $220M in university base funding by 2009-10 to substantially expand graduate education, adding 12,000 full-time graduate student spaces by 2007-08, and 14,000 by 2009-10. It is difficult to see how the CGS on its own could increase national enrolment as a whole given that enrolment in general is always larger than the number of scholarships that are available in the first place. However, this does not preclude the synergy of scholarship availability with education funding by the provinces, the facilitative effect of financial support in terms of the students’ ability to devote more time to their studies, and the power of attraction for high caliber students who are the ones who may choose to study essentially anywhere in the world.

Figure 5: Increase in Graduate Enrolment Over Time in Canada

Source: Statistics Canada.

2.1.3 Increase incentive for recipients to complete studies within specific time period

For key informants, the CGS award funding level is seen as supporting students so they can spend more time on research as opposed to working part-time to support their studies but it was felt that there is no clear indication that this results in more timely completion of graduate degrees. Other factors were seen as slowing progress more so than the amount of funding (i.e., research not going well; personal reasons). In addition, supervisors also felt that the duration of the CGS award did not create more of an incentive to complete studies than did other awards or funding sources. It was noted that all students appear to be more motivated if their funding is running out.

The views of focus group participants suggest that the CGS facilitates the completion of studies in a timely manner defined as completing studies in less time than they probably would have without a CGS. However, no participants felt it was possible to complete their degree during the period of the CGS (i.e., one year for a master's and three years for a PhD). Recipients often spoke about how the CGS allowed them to focus exclusively on their studies, unfettered by the need to serve as a teaching or research assistant or by having to work outside the university. They also often highlighted the psychological benefits of being able to count on a living wage for an extended period of time, to which, in turn, they attributed an increased likelihood that they would finish on time.

“Completing the program would have been more difficult and longer. I would have had to work as a TA and spend less time on research.”

“Receiving a CGS allowed me to choose the type of teaching assistant work I did.”

Similarly, many of applicants felt that having a CGS would very likely have allowed them to complete their studies in a timelier manner. In this regard, some master's students lamented the negative impact that working outside the university had had on their time and energy. There was some sense among applicants that taking longer and having more financial stress could very easily affect the quality of their studies. Some spoke about having terminated their research early in an effort to limit expenses.

“I thought I could finish on time, but I was working 35 hours a week. When you get home you just don’t have the energy to do much.”

“There is no question it’s going to take me longer than it would have with a Bombardier [CGS].”

Consistent with the expectations expressed above, the survey data show that, regardless of duration of the studies, CGS recipients (78.8%) are more likely than applicants (68.5%) to complete their studies. Students were asked whether they had completed their study program (with a Yes/No response) and those who said no were asked a follow-up question whether they were still studying in their program (Yes/No). Students who answered no to both questions were then asked whether they had withdrawn voluntarily, being asked to withdraw or had no response. Withdrawal rates for CGS recipients (2.2%) are lower than those of applicants (7.0%) and the gap is larger for doctoral students (1.6% recipients vs. 7.7% applicants) than master's (2.6% recipients vs. 5.4% applicants). The highest withdrawal rates are observed among scholarship applicants enrolled in master's programs in HS (9.7%) and among applicants enrolled in doctoral programs in SSH (9%) as shown in Figure 6.

Figure 6: Percent of Scholarship Recipients and Applicants Who Withdrew from their Program, Voluntarily or Not

Source: Surveys of CGS Recipients and Applicants.

To analyze the extent to which CGS provides incentives to complete a degree program faster, the surveys inquired about the impact that the CGS had on the pace of progress and recipients’ circumstances such as their financial situation and quality of life. The pace of progress was further explored in terms of reasons for being ahead or falling behind of plan to complete the program of studies. The analysis also identified the specific attributes of the CGS awards that may contribute to faster program completion.

On average, having a CGS scholarship is reported to have a positive impact on many aspects of a recipient’s experience, including the pace of their progress through their study program (5.7 average), their quality of life and their need for financial support (Table 5). Regarding the incentive to complete their program of studies, CGS recipients consider, on average, that the award had a slight positive/positive impact on the pace of their progress through their study program. The impact is more positive for recipients enrolled in a doctoral degree (5.9 average) or in SSH (6.0 average). Furthermore, the scholarship had a positive impact on the time available to devote to their studies (6.3 average) which is expected to contribute to timely completion of their program.

CGS recipients also report positive impacts of the scholarship on their financial situation during their studies (6.6 average), on the quality of life during their studies (6.5 average), on the need for obtaining income during their studies (6.3 average) and on their current financial situation (6.1 average). This positive assessment is greater for doctoral and SSH recipients.

Table 5: Average Impact of CGS Awards, CGS Recipients

Analyses of other aspects of the pace of progress indicate that on average, CGS recipients are progressing according to plans or better more than applicants (3.5 and 3.2 average respectively^{Footnote 13}). For both groups, the extent to which they are meeting their plans is different according to their degree level and their area of study (the difference in means is statistically significant), where students in doctoral programs and in SSH report advancing according to plans/a little ahead more often. The better rate of progress according to plans reproduces a similar observation made in 2008.^{Footnote 14} In addition, continuity regarding the research topic and the supervisor are often cited as reasons for being ahead of the plan regarding progress in the program. On the other hand, personal reasons and a wide category labelled research/thesis taking longer than expected are the reasons more often cited for falling behind.

In the previous section, it was noted that issues of a financial nature play a role in the pace of progress through the program, among other reasons. It was also noted that lack of funding impacts applicants more negatively than CGS recipients. The surveys inquired about the extent to which applicants and recipients were concerned with their financial situation prior to applying to their program of studies. Both groups report being moderately concerned on average, with applicants being more concerned than recipients.

The evidence regarding loans indicated above is confirmed with the distribution of average debt (Table 6). Across all degree levels and areas of study, the majority of recipients have no debt accumulated (66% to 82%) whereas among applicants the proportions range from 42% to 62%. Furthermore, a greater proportion of applicants than recipients report debt in all ranges. The advantage of recipients over applicants in terms of quantum of debt accumulated was also reported in the SSHRC Doctoral Fellowship evaluation where both CGS and SSHRC Doctoral Fellowship recipients had lower mean debts over the periods of their degree ($6,390 vs. $6,114) than applicants ($11,005).^{Footnote 15} The NSERC Scholarships and Fellowships evaluation reported similar results at the doctoral degree level: their CGS ($2,660) and PGS ($3,590) recipients had significantly lower mean debt loads than applicants ($8,330).^{Footnote 16} The fellowship with the lowest accumulated debt is, not surprisingly, the Vanier CGS Doctoral Fellowship where 85% reported no debt and 8.9% had more than zero but less than $20,000.^{Footnote 17}

Table 6: Distribution of Average Debt Accumulated

Correlated with greater debt and loans, applicants report on average, that they worked at a non-academic job for pay somewhat by necessity (3.1 average for applicants) rather than by choice(4.3 average for recipients).^{Footnote 18} A similar trend was observed in the SSHRC doctoral fellowship evaluation where doctoral fellows (9%) and CGS recipients (7%) in the 2008-2011 cohort reported working in a non-academic position for pay by necessity somewhat, mostly or only more often than applicants (36%).^{Footnote 19}

The surveys also inquire about how many hours per week both recipients and applicants dedicate to paid and unpaid activities. There were no differences in the number of paid hours dedicated to fulfill the requirements of the degree program (see Table C1 in Appendix C). There are also no statistically significant differences regarding the number of paid hours dedicated to other research and/or teaching activities. However, recipients spend reliably less paid hours than applicants on paid non-academic employment, especially at the doctoral level, and in SSH. The SSHRC Doctoral Fellowship evaluation reported a similar trend of spending the least number of paid hours on non-academic employment, followed by research and teaching activities outside degree requirements and work to fulfill degree requirement among the more recent cohort of students (2008-2011).^{Footnote 20} A similar trend was observable in the NSERC evaluation.^{Footnote 21}

No statistically significant differences are observed between applicants and recipients regarding the number of unpaid hours dedicated to fulfill the requirements of the degree program nor research and/or teaching activities (see Table C2 in Appendix C). However, applicants at the doctoral level in SSH spend reliably more hours than CGS recipients on unpaid non-academic employment.

2.1.4 Increased recognition by the research community of the federal government’s financial support for research training

According to all key informants, rather than increasing the recognition of the federal government’s financial support for research training on its own, the CGS program does so as part of a portfolio of awards from the federal granting agencies. Informants felt that the agencies have an international reputation for supporting research excellence yet none gave specific evidence of this international reputation.

Universities appear to be well aware of the CGS program as a result of communications from the three federal granting agencies. Faculty and administrators involved in research make efforts to be up-to-date on the CGS program along with other awards. This is supported by the focus group participants who suggested that the CGS is well known among graduate students and that universities, as well as individual faculties and departments, are devoting resources and efforts to raising awareness of the CGS and other funding opportunities among their students. Recipients and applicants alike identified a range of ways in which they and their friends learned about the CGS:

• Professors
• Workshops and seminars organized by a university or department
• Posters
• Other students (e.g., those a year or two ahead)
• Mass e-mails sent by the university or department
• As part of the application process to graduate studies

There was an impression held by many students that universities, departments, and even professors, had an interest in helping as many students as possible to obtain scholarships from outside the university because more funding from the government would "save money for the university".

“If you are getting your funding from the government, you don’t need to get it from the university. It saves them money.”

A similar perception was evoked by a key informant who thought that the CGS program could unintentionally influence universities' decisions on the acceptance of candidates due to tight university budgets whereby more CGS recipients could mean less need for the university to provide its own funding.

“The university is forced to accept from the pool of candidates each year rather than turn money back. Under the former process, we had the option of not recommending candidates. In some years the pool may not be as strong. It would be good to be able to carry forward some awards to the next year to ensure a better use of the funds for good students.”

This attribution is at odds with the widespread awareness among the students that they are generally required to apply for external funding before being considered for institutional funding, thus implying that external scholarships allow universities to augment the amount and/or the number of institutional scholarships they provide.

Administrators and supervisors had the following suggestions for ways of increasing awareness of the CGS, which would be a combination of efforts by granting agencies and universities:

• Simplify the naming of the CGS awards
• Reach out to all researchers including those doing less research
• Reach out to undergraduates
• Advertise throughout the whole year instead of only as part of the application cycle
• Keep award websites up to date
• Continue to clarify rules and processes under harmonization
• Develop a PowerPoint presentation that could be used by universities

The suggestion to simplify the naming was supported by the students in the focus groups. Very few participants referred to the CGS as the "Bombardier", "Bell" or "Banting-Best" Scholarship, though there was some degree of recall of these names. More often than not, the scholarships were referred to by the granting agency name (e.g., "the CIHR" or "the NSERC" award). It was clear, however, that students were fairly knowledgeable, and knew, for example the difference between the CGS and the other agency awards/scholarships. In addition, the CGS recipients do know that it is funded by the federal government and make the distinction with those offered by the provincial government or the university.

The suggestion of publicizing the scholarships more at the undergraduate level was also echoed in the focus groups where both recipients and applicants indicated that the CGS, and scholarships in general, are much less well-known at the undergraduate level due to a lack of systematic promotion of the award at that level. Participants who had been aware of the CGS as undergraduates said that they usually came across the information through informal channels, most often, word of mouth.

“It’s not publicized. In my case as an undergraduate, it was a professor who came up to me and told me about it. Otherwise I would not have heard about in the second year of my BA.”

“If during your undergraduate degree you work 9 to 5 and don’t participate in the activities and social life, then there is a good chance that you won’t hear about it.”

2.2 Intermediate Outcomes

The program's intermediate outcomes are to:

• Increase the number of students completing degrees and doing so in a timely manner
• Provide high quality research training, as well as increase the ability to attract and retain experienced researchers
• Increase the capacity to meet the demand for HQP in Canadian universities and in the public and private sectors
• Improve the branding of Canada as a home of research excellence and Canadian universities as world-class research centres

2.2.1 Increase the number of students completing their degrees and doing so in a timely manner

In the preceding section on immediate outcomes, both key informants and students had indicated their perception that the CGS can act as an incentive to complete studies, reasoning that the scholarship provides a means of living for a limited duration of time. There was consistent evidence of such an incentive effect in the survey data where recipients were observed to withdraw less than applicants. Returning to the issue of completion and its timeliness as an intermediate outcome, much of the discussion in the focus groups revolved around the issues of amount and duration of the CGS award.

Among recipients, the views of master's students concerning the amount they received from a CGS tended to differ somewhat from those of PhD candidates. There was general agreement among CGS doctoral recipients that the $35,000 per annum non-taxable amount they received was very good; that it allowed them to live comfortably while devoting themselves entirely to their studies, even in cities such as Vancouver where rents can significantly exceed the national average. This level of yearly funding was seen as quite sufficient, particularly when coupled with additional funding from a supervisor and/or travel funding from the MSFSS. Master's recipients also saw the $17,500 amount of their CGS as very significant and helpful, but they also could not help contrasting it with the much higher amount that the doctoral students receive. There was agreement that it was very difficult for a master's student to live autonomously on the CGS. Most of applicants also felt that the amount required should be identical regardless of whether they were pursuing a master's or doctoral degree.

It is clear that students tend to assess the value of the scholarship not as a complement to other means of living but rather as the main source of financial support. Applicants who participated in the focus groups also expressed that an amount of $25,000 to $35,000 per annum would be sufficient for a graduate student to study full time, while enjoying a reasonable standard of living. Applicants who had not been able to obtain much other funding spoke about how they had to rely on student loans, a spouse, moving back with their parents, or earning money by working off campus. Yet participants did not get into the specifics of what they owed or had to borrow or the impact on their progress and grades; rather they spoke of the stress it caused them and the time it took away from their studies.

Some participants noted that the adequacy of the amount might vary depending on the cost of living in the city where a person lived, as well as on their family circumstances. Quite a few participants in the Montreal focus groups, including both CGS recipients and applicants, wondered whether number of dependent children could be taken into consideration when awarding the scholarship, possibly by topping-up funding. The survey data indicate that 28% of respondents have one or more dependents. In a similar vein, some of the parents in the Montreal groups spoke about the paradox they saw in the fact that the CGS was, for all intents and purposes, meant to replace a job, but because the money they received could not be considered income, it prevented them from qualifying for certain benefits, such as parental leave and other programs and benefits that are determined by considering the income of the applicant, even though the money was taxed by the provincial government.

The ability of applicants to leverage enough money to complete their studies in a timely manner while enjoying a satisfactory standard of living varied significantly. A few suggested that the amount of the PhD award was too high, considering that it could be paired with a stipend and waived tuition in some cases, but most stated that they were struggling. There were a few students who had obtained university scholarships coupled with well-paying research contracts from professors. In these cases however, it was felt that their program would not be completed as soon as they had hoped because the contracts still involve an amount of work taking time away from their program of studies.

Views on the duration of the CGS were fairly consistent across the six focus groups. The initial reaction of many was that it was somewhat counterintuitive for such a prestigious scholarship not to correspond to the amount of time that is typically required to complete a program of study (i.e., two years for a master's and at least four for a doctoral degree versus the award duration of one year for the CGS-M and up to three years for the CGS-D). Some supported their view by explaining how the most stressful and time-consuming part of completing a graduate degree comes in the final stages when CGS may have run out. Recall that this effect was cited as an incentive to complete early by one key informant (out of 17).

There was no suggestion that fewer scholarships should be granted in order to allow the same annual amount of funding to be available over a longer period. In one SSH focus group, a participant suggested that the Vanier CGS be reduced in amount in order to increase the funding for CGS awards, but reactions to this were mixed, mainly because participants felt that with the limited number of Vanier CGS awards available, the impact on the CGS program would be small.

Program completion rates, continuation rates and average duration based on survey data provide empirical information regarding the impact of the CGS award on timely completion of their program. Students were asked whether they had completed their study program (with a Yes/No response) and those who said no were asked a follow-up question whether they were still studying in their program (Yes/No). Students who answered no to both questions were then asked whether they had withdrawn voluntarily, being asked to withdraw or had no response. Among doctoral students overall, CGS recipients (64.4%) report higher program completion rates than applicants (59.9%) but there are no significant differences between CGS recipients and applicants at the master’s level.

Table 7: Program Completion, % CGS Recipients vs. Applicants

The majority of doctoral-level (95.2%) and master’s level (60.9%) CGS recipients who have not yet completed their studies report they are still enrolled in their program, a greater proportion than reported for doctoral (79.5%) and master’s (43.4%) applicants respectively (see Table C3 in Appendix C). At the master's level, the difference in continuation rates between CGS recipients and applicants is likely driven by students enrolled in SSH but at the doctoral level all three study areas appear to have an impact.

Regarding program duration (Table 8), CGS recipients report that they complete their program one trimester faster than applicants. This difference is primarily for doctoral recipients in SSH and NSE. Not at all surprisingly, program duration overall differs significantly by degree level – those at the doctoral level take at least twice the time to complete than master's – and by area of study where average duration for both master's and doctoral levels ranges between three and a half and four years.

Table 8: Average Number of Months to Complete Degree, CGS Recipients and Applicants

Further analysis of program duration in terms of cumulative proportions that completed their program within a set time frame, confirms the absence of any differences between CGS recipients and applicants at the master’s level but it is very clear that doctoral recipients complete their programs faster than applicants at the doctoral level (Figure 7). It also confirms that doctoral programs take longer to complete than master’s as noted in the focus group reports.

Figure 7: Cumulative Proportion Completing Study Program by Duration and Degree Level, CGS Recipients and Applicants

Source: Surveys of CGS Recipients and Applicants.

Detailed data on program duration disaggregated by year of entry into degree program (cohort) for both applicants and recipients are presented for master's program and doctoral programs in Appendix C (Tables C4-C7). In reading these tables, it is important to remember that the more recent cohorts have not had enough time for their duration distribution to unfold. Nevertheless, the pattern among the cohorts that have had sufficient years to complete confirms the focus group reports that master's last longer than one year, typically two or three years, and doctoral level last longer than three years, typically up to seven or more.

Considering the 2002-2011 cohorts the SSHRC and NSERC scholarship evaluations did not find any significant differences in months to degree completion among doctoral trainees irrespective of funding status.^{Footnote 23} Similarly, there were no observable differences between CGS-D and Vanier CGS recipients.^{Footnote 24}

2.2.2 High quality research training and increased ability to attract and retain experienced researchers

Both administrators and supervisors felt that CGS recipients and applicants are provided with high quality training by their supervisors. Students in focus groups stated that this is how it should be. However the amount of the CGS and/or its combination with other funding (e.g. institutional awards or research assistant remuneration) are seen by key informants and both recipients and applicants as enriching opportunities to attend conferences, network with other researchers and work in other laboratories to collaborate on research. Some CGS recipients can also access the MSFSS which opens opportunities to do international research.

Some supervisors and administrators felt the CGS award may impact a university’s capacity to attract and retain highly qualified researchers. Specifically they noted that having CGS award winners and the kinds of quality research they do may enhance the reputation of a university and make it attractive to highly qualified researchers who then supervise these award winners. At the same time, some supervisors felt that the presence of highly qualified researchers (supervisors of graduate students) at a university attracts the top students (including CGS award winners or applicants). This is made explicit for professors seeking NSERC grant funding as they get credit for including graduate students on their application. In this sense the two sources of funding reinforce each other. In addition, recipients agreed that professors were more likely to take an interest in a CGS recipient or the recipient of other substantial funding and even that some professors will not accept to supervise students unless they have funding.

Some administrators commented that both researchers and prospective graduate students notice how many CGS awards are granted to students at a given university and interpret this as a sign of its research excellence. Some universities also profile their CGS recipients as part of their overall research communication strategy. For most recipients, this boosted their confidence and, for some, had an indirect impact on the quality of their education through an enhanced level of motivation and self-esteem as a scholar.

Recipients in focus groups indicated that the positive impact of the CGS on the quality of their training came through having more time to focus and produce higher quality work, being able to afford to travel and purchase equipment, and having greater choice in the teaching and research assistantships they might take.

Several factors influence the quality of the research training, including the research environment, interactions with the supervisor and other researchers, and involvement in research and other academic activities. These opportunities are reflected in the production of research outcomes and in the extent of improvement in research abilities and other skills. The surveys provided evidence for many of these elements.

The research environment is directly impacted by the quality of the supervision and the availability of equipment and infrastructure to guide and pursue the research program. Satisfaction levels were generally high typically ranging from a little satisfied to satisfied. CGS recipients report being more satisfied than applicants with the supervision provided by the primary or co-supervisor and the equipment/infrastructure available and these differences appear to be driven mainly by recipients in SSH (Table 9).

Like the CGS, the SSHRC and NSERC evaluations also reported high levels of satisfaction with the research environment irrespective of funding status. The NSERC evaluation reported 73% to 91% were satisfied among the master’s level respondents and 82% to 89% among the doctoral level respondents.^{Footnote 25} Over 85% said they were a little satisfied, satisfied or very satisfied among the SSHRC recipients.^{Footnote 26}

Table 9: Average Satisfaction with Environment of Graduate Program, CGS Recipients and Applicants

Interactions with their supervisor and other researchers are important contributors to the quality of training and the research experience. One-third of CGS recipients report interacting with their supervisors several times a week, while close to one-fifth report weekly interactions and over one-fifth report interactions with their supervisor several times a month. The same pattern is observed at the master's and doctoral levels. Interactions with other researchers in the same discipline in Canada are also quite common, with close to half of doctoral students and approximately 40% of master's students reporting such interactions between several times a week and several times a month. Over 90% of CGS-D recipients and approximately 70% of CGS-M recipients report having interacted with researchers in their discipline outside Canada. Interactions with researchers in other disciplines in Canada are common for both degree levels, but less frequent. Doctoral recipients had more frequent interactions than master's, primarily in the once to several times a month range with researchers in and outside their discipline in and outside Canada.

Interactions by area of study indicate that interactions with supervisors are common, but tend to be less frequent in SSH. Interactions with researchers in same discipline in Canada appear to have the same frequency for all areas of study. On the other hand, CGS recipients in HS report having more interactions with researchers in their discipline outside Canada and with researchers in other disciplines in Canada and outside Canada. Recipients in NSE and in HS had more interactions than SSH recipients with all categories of researchers.

Another aspect of the quality of training is the extent of involvement in research-related activities (Table 10). There are several areas in which CGS recipients report greater involvement than applicants, including the data collection phase/research implementation, the development of research protocol methods, and preparing research/funding proposals.

Table 10: Average Level of Involvement in Research-Related Activities, CGS Recipients and Applicants

Consistent with findings from the focus groups, CGS recipients indicate being more involved in supervising other students than applicants (4.8 average vs 4.4) while, on the other hand, applicants report having more involvement than recipients in developing course materials (4.2 average vs 3.9) and actually teaching university courses (4.4 average vs 3.9).

Involvement in service-related activities, such as administrative roles, participation in student groups, volunteer organizations and other civic initiatives are generally less common than involvement in either research or teaching related activities across degree levels. However, CGS recipients at the doctoral level are more likely to take on an administrative role or a leadership position in a student group or student organization than master’s students.

CGS recipients and applicants have similar behaviors regarding their involvement in professional/personal activities. The activities more commonly mentioned as part of their graduate program experience are critical and creative thinking, interpersonal communication and networking and collaboration. Applicants report having been involved more often in critical and creative thinking than CGS recipients.

Both CGS recipients and applicants are generally satisfied with the opportunity to develop their research, teaching, civic and personal/professional skills with the greatest level of satisfaction observed for research-related activities, particularly for CGS-D recipients (6.4 on average on 7 point scale).

An expected outcome from interaction with the supervisor and other researchers, as well as the opportunity to participate in research-related activities is that research skills be improved. Both CGS recipients and applicants concur that the vast majority of their research-related skills improved either noticeably or significantly. Improvement in teaching-related activities is not as significant as for research. Both CGS recipients and applicants agree in general with the extent of improvement. In particular, areas of improvement include communications/presentations and supervising other students. Improvement in professional/personal skills is also quite similarly reported by both CGS recipients and applicants. It is interesting to note that although applicants reported greater involvement in critical and creative thinking activities, their level of improvement reported is equivalent to that of CGS recipients (5.6 for applicants and 5.4 for recipients on a 7 point scale, not statistically different).

The productivity of graduate students, measured in terms of publications, presentations, patents and other outcomes, is another measure of the quality of their training. On average, CGS recipients produce a greater number of published articles (average 3.2) and other research and academic papers (average 1.6) and give more presentations at conferences (average 7.6) than applicants (averages of 2.0, 1.2 and 6.0 respectively, Table 11). These results are primarily driven by CGS recipients enrolled in doctoral programs and in HS disciplines as they have more peer reviewed publications and conference presentations including international presentations. The higher productivity of the doctoral CGS recipients in the same types of outputs parallels the 2008 CGS evaluation report. In the current evaluation doctoral recipients report an average of 4.7 articles and 10.8 conference presentations, values which are double those reported in the 2008 evaluation (average of 2.3 and 5.6 respectively).^{Footnote 27}

Table 11: Average Number of Publications and Other Outcomes by Degree Level and Area of Study

In the agency-specific evaluations, there were no differences in productivity between SSHRC Doctoral Fellowship recipients and CGS recipients but the SSHRC Doctoral Fellowship recipients were more productive than their corresponding cohort of applicants regarding peer reviewed articles, research papers, books and book chapters, conference presentations and presentations at international conferences.^{Footnote 28} Among NSERC respondents, CGS-D recipients reported more peer reviewed publications than PGS, IPS/IIS, CREATE as well as applicants. Amongst master’s level recipients, CREATE recipients outperformed other funded groups and applicants in several productivity categories including peer-reviewed articles, oral or poster conference presentations, art exhibitions, research tools, patent applications files and patents granted and other intellectual property claims filed.^{Footnote 29}

Additional specific measures of productivity are presented in Table C8 (Appendix C). Recipients differ significantly from applicants on the majority of the outputs although the direction of the difference varies depending on the nature of the output. The most common output is findings being cited by others (17.4% of recipients at the doctoral level and 14.2% of recipients at the master’s level). This is closely followed by a new theory or a new research method emerging from the research. “Findings cited by others” is the most common output across all fields of study. Other outputs differ by area of study. In SSH, new theory (12.4%), new research method (11.2%) and plain-language summaries (9.2%) are next most frequent. In NSE, it is professional practice (14.9%), adaptation of research findings (11.6%) and new research method (9.6%) and in HS it is media (10.5%), professional practice (10.3%) and new research method (8.4%).

Students with high-quality research training may be inclined to pursue a postdoctoral fellowship. CGS recipients report that they are moderately interested in pursuing a fellowship (average of 5.1 out of 7). For those who have already graduated from a doctoral program, CGS recipients (0.6) in SSH report a higher number of postdoctoral fellowships held than their applicant counterparts (0.4).

2.2.3 Increased capacity to meet the demand for HQP in Canadian universities and in the public and private sectors

Evidence from the survey suggests that holding a CGS is having a positive impact for recipients in terms of gaining full time employment, particularly at the doctoral level, and in SSH (Figure 8). Doctoral recipients in both the earlier (87.7%) and more recent (88.0%) cohorts were more likely to be working full time as compared to their respective applicant cohorts (79.6% in the earlier cohort vs. 63.8% in the more recent cohort). Similarly, recipients in SSH were more likely to be working full time than applicants in both cohorts (81.7% vs. 77.4% for the earlier cohort and 74.9% vs. 64.5% for the more recent cohort). The SSHRC evaluation reported a similar advantage for Doctoral Fellowship recipients over applicants however, they did not observe any significant difference between the SSHRC CGS and DF recipients.^{Footnote 30} There were also no significant differences in full time employment status among CGS doctoral recipients and NSERC agency-specific recipients in the NSERC evaluation.^{Footnote 31}

Figure 8: Percent Employed Full Time by Study Area and Degree Level by Cohort, Recipients vs. Applicants

Source: Surveys of CGS Recipients and Applicants.

Cohort 1: 2002-2007; Cohort 2: 2008-2011.

Both applicants and recipients report that their degree program was useful training for their career (average 5.9 on a 7-point scale). In addition, both groups revealed that their current employment was moderately to very much related to their degree program. On average, CGS recipients (5.7 on a 7-point scale) were more likely to report that their employment position is closely related to their degree program, compared to applicants (5.2). The relationship held true among master’s students (recipients – 5.3 vs. applicants - 4.8) and doctoral students (recipients – 6.1 vs. applicants - 5.7). These findings replicate the 2008 evaluation results where 80% (master’s) to 93% (doctoral) recipients judged their employment to be related to their graduate studies compared to 65% to 88% respectively for applicants.^{Footnote 32}

Most graduates working in positions that were less related to their degree program noted that they did so due to the unavailability of jobs in their fields of study. There were few notable differences between applicants and CGS recipients across degree levels. CGS-D recipients were more likely to indicate that a change in career or professional interests influenced their decision to work in an area less than somewhat related to the field of their degree program (23.3%), compared to applicants at the same degree level (15.3%). This was also true of recipients and applicants across the three areas of study. Furthermore, applicants in HS were more likely (23.8%) to indicate that pay and promotion opportunities was an important factor compared to HS recipients (11.1%).

The student survey did investigate perceptions of job readiness and acquired skills in terms of research, teaching, service to the academic community, service to the larger community, and personal/professional domains. Respondents reported being a little satisfied to satisfied in most of the categories. Overall, respondents were most satisfied with opportunities to develop research skills and personal/professional skills and CGS recipients were more satisfied with opportunities to develop skills than applicants (Table 12).

Table 12: Average Satisfaction With Opportunities To Develop Skills During Graduate Studies, CGS Recipients Vs. Applicants

CGS recipients were asked about the contributory factors to obtaining their current employment and among currently employed recipients, personal and professional experience, and research-related experience were the most commonly identified factors which they believe helped them obtain the position they currently hold (Table 13). There were a few notable differences in perceptions across degree levels and areas of study. For example, doctoral recipients were more likely (31.8%) to indicate that their research-related experience helped them obtain their current position compared to master’s recipients (22.8%). Across areas of study, respondents in SSH were less likely to identify research-related experience (22.4%) compared to respondents in the two other areas. Respondents in SSH however, were more likely to identify their experience serving their larger community as a factor (11.7%).

Table 13: Contributory Factors to Obtaining Current Employment, CGS Recipients

In connection with job readiness, the analysis showed that the university sector was the most frequently mentioned intended employment destination out of the four sectors. It was the preferred sector for about two-fifths of doctoral recipients, doctoral applicants and master’s recipients but only just over a quarter of master’s applicants (see Table C9 in Appendix C). It is interesting to note that the three other sectors (private, government and not-for-profit) combined were still the intended destination for the majority (three-fifths) of the students and there were no significant differences between recipients and applicants.

The reality of employment (Figure 9) presents a similar picture across degree levels and areas of study, with lesser proportions of the 2008-2011 cohort (labelled C2 in the chart) working in the university sector as compared to the 2002-2007 cohorts (C1). Doctoral graduates are more likely to be employed in the university sector while master’s graduates are more likely to work in the private or government sectors. Among the doctoral graduates, recipients are more likely than applicants to be employed in the university sector, more so for the 2008-2011 cohort (68% recipients vs. 59% applicants).

The NSERC evaluation found doctoral respondents to be more likely to be working in universities while the master’s level respondents were more likely to be in the private sector.^{Footnote 33} In the SSHRC evaluation, across both cohorts, there were no differences between SSHRC Doctoral Fellowship recipients and CGS recipients in likelihood of working in the university sector.^{Footnote 34}

Figure 9: Sector of Employment by Cohort, CGS Recipients and Applicants

Source: Surveys of CGS Recipients and Applicants.

A = Applicants; R = Recipients; C1 = 2002-2007 cohort; C2 = 2008-2011 cohort.

The positive impact of the CGS is particularly apparent among CGS recipients employed in universities and having studied at the doctoral level. CGS-D recipients are more likely to hold research-faculty, scientist, associate, fellow or similarly titled positions than applicants, an effect most visible in the 2002-2007 cohort (40.8% recipients vs. 34.0% applicants) which has reached more maturity in terms of employment as compared to the 2008-2011 cohort (18.9% recipients vs. 11.8% applicants; see Table C10 in Appendix C). The 40.8% of CGS-D recipients holding a research faculty position reported here is higher than the 13% reported in the 2008 CGS evaluation^{Footnote 35}, again reflecting the maturity of the employment outcomes for this cohort. Doctoral recipients are also more likely to hold postdoctoral positions than doctoral applicants (18.7% vs. 13.4% for 2002-2007 cohort and 54.0% vs. 28.1% for the 2008-2011 cohort respectively).

The benefits of holding a CGS seem to also extend outside of academia. Among respondents employed in other sectors, CGS recipients at the master's level were more likely to indicate that they were employed as a researcher compared to applicants in the 2002-2007 cohort (see Table C11, Appendix C). This was true for both cohorts at the doctoral level. In addition, the proportion of employment as a researcher appears highest in the HS followed by NSE and SSH, in that order. Applicants at the doctoral level reported being in junior-level positions more frequently than CGS recipients for 2008-2011 cohort. This was also true of applicants and CGS recipients in the SSH (2002-2007 cohort), and in HS (both cohorts).

Apart from skills, two basic conditions for the CGS program to increase the capacity to meet the demand for HQP in Canadian universities and in the public and private sectors are that many students complete their degrees and stay in Canada afterwards.

As previously noted (see Section 2.2.1) survey findings reveal that holding a CGS may influence degree completion at the doctoral level. CGS doctoral recipients were significantly more likely (64.4%) to indicate that they had completed their doctoral degree compared to applicants (59.9%) and the difference was strongest in the HS area. CGS recipients who had not completed their studies yet were also more likely to continue their degree program, both at the master's and at the doctoral level.

Survey respondents overwhelmingly chose to remain in Canada after completion of their studies. In total, 85.7% of respondents reported currently living in Canada. While there was not much difference between recipients (88%) and applicants (91%) at the master’s level, recipients at the doctoral level (78%) tended to be less likely than their applicant counterparts (85%) to remain in Canada. Conversely, CGS recipients (all degree levels and areas) who were not in Canada at the time of the survey expressed a stronger desire (5.4 on a 7 point scale) than applicants (4.3) to return to Canada in the future to pursue employment opportunities.

2.2.4 Improved branding of Canada as a home of research excellence and Canadian universities as world-class research centres

Administrators felt that overall the CGS is a prestigious award, on a par with some other national awards (i.e., awards of federal granting agencies, Trudeau, Killam), above the stature (and dollar value) of provincial awards, higher value than most institutional awards, and more accessible than the few international awards (e.g., Rhodes). Supervisors similarly felt that the CGS award has increased their university’s reputation for research excellence in that the award is considered prestigious and granted to top students. Both recipients and applicants viewed the CGS as among the top scholarships in Canada, second only to the Vanier Scholarship.

Both recipients and applicants agreed that the CGS was not known outside of academia. With respect to the non-academic labour market, they felt that, in the eyes of most employers, the CGS would be indistinguishable from lesser scholarships and grants.

Survey findings suggest that the CGS is positively impacting the output of recipients and is contributing to branding Canada as a home of research excellence. Overall, compared to applicants, CGS recipients are significantly more likely to have published more articles in peer-reviewed journals (3.2 vs. 2.0 average), authored or co-authored more research papers, books, book chapters and publications (1.6 vs. 1.2 average), and to have given more oral or poster conference presentations (7.6 vs. 6.0 average), particularly at international conferences (3.5 vs. 2.7 average). As could be expected, this is particularly true of CGS recipients at the doctoral level who outperformed applicants across all of the outputs previously cited. Across areas of study, CGS recipients in NSE (3.3 vs. 2.1 average) and HS (4.8 vs. 2.8 average) are outperforming applicants in publishing articles they have written or co-written in peer-reviewed journals. Recipients in HS also reported giving more oral presentations (10.9 average) than applicants (7.6 average), and doing so more often at international conferences (5.0 vs. 3.2 average).

2.2.5 Unanticipated outcomes

Administrators and supervisors felt that the duration of the award may work against the CGS objective of being an incentive to undertake graduate studies, causing students to delay applying until into their program. There is evidence of delay in applying for the scholarship in the survey data showing that up to 40% of recipients enrolled before they obtained the award but they did not delay enrolment per se. It may be that the cause for the delay in applying for the scholarship is not its duration but a lack of knowledge of the existence of the CGS at the undergraduate level. It could also be that they are deliberately holding off in order to have more competitive applications the further along their degree they are, and/or have the funding get them further along their graduate studies.

One informant expressed the concern that because universities must use all the allocated CGS-M funds in the year they may select less than ideal candidates to fill their quota. Given the very high selectivity of the scholarship competition in general (93% of recipients have undergraduate GPAs in the A-, A, A+ range), this concern would appear to be a rare occurrence.

In the intermediate term, a few administrators suggested that program recognition in terms of brand may actually be hindered by the variety of specific names rather than using the single expression "CGS award" throughout.

There is a perception among non-recipients that the selection process is not transparent and also, presumably at equal qualifications, that a CGS doctoral award is easier to obtain if a student had received one at the master's level. The latter view was also expressed by administrators and supervisors.

2.3 Exploration of Long-term Outcomes

Long-term outcomes are studied in terms of the perceived impacts of the CGS program on the career path and the achievements of CGS recipients following their experience with the program (e.g. obtaining other grants or award funding, presentations at conferences, production of papers, books, book chapters, etc., employment earnings). Impacts on international exposure are also examined.

2.3.1 Impact on potential career path of recipients

In the focus groups, CGS recipients reported that the impact of the CGS on their career path was to generally validate their decision to pursue a career in academia and seen to facilitate this career. The impact on non-recipients, especially at the master's level was that some who had once aspired to academia were revaluating their options in light of their lack of success at obtaining a CGS. In addition, because obtaining a CGS, even at the PhD level, is by no means a guarantee that recipients will become professional scholars/researchers, many recipients evaluated opportunities outside academia, including in the private sector mostly for NSE and HS.

In the survey, the perceived impact of the CGS Program on career plans was that CGS recipients (5.7) were more likely to be currently employed in a position that is closely related to their degree program, compared to applicants (5.2). Simultaneously, both recipients (26.3%) and applicants (27.1%) who were employed in positions that were less related to their degree program attributed the reason most often to the unavailability of jobs in their field.

The achievements of CGS recipients are numerous. As stated before, they produce on average a greater number of published articles (3.2) and other research and academic papers (1.6) and give more presentations at conferences (7.6) than applicants (2.0, 1.2 and 6.0 respectively). These results are primarily driven by CGS recipients enrolled in doctoral programs and in HS disciplines.

CGS recipients report greater involvement than applicants in five out of 11 research-related activities: preparing research proposals (recipients 5.0 vs. applicants 4.8); development of research protocol/methods (5.7 vs. 5.4); data collection/research implementation (6.2 vs. 6.0); knowledge of research integrity/ethical conduct (4.9 vs 4.6) and leading research projects including human, financial and time management (4.4 vs. 3.8). CGS recipients at the doctoral level are more likely to take on an administrative role (3.8 vs. 3.2 for master’s) or a leadership position in a student group or student organization (3.4 vs. 3.1) than master's students.

There are no significant differences observed in the number of awards obtained by both groups of students. The awards/prizes most commonly obtained are from the institution where the program is being pursued (2.5 on average for CGS recipients); this is followed by awards/prizes from a Canadian provincial body (0.6 on average for CGS recipients). In terms of their average value, research stipends/assistantships paid from a research grant from CIHR, NSERC or SSHRC are significantly larger for CGS recipients ($30,664 vs. $19,873 for applicants, $47,400 for recipients at the doctoral level (vs. $18,101 for doctoral level applicants). Note though that master’s level recipients report a lower amount ($13,926) than master’s level applicants ($21,645). Applicants, on the other hand, obtain teaching assistantships with a greater financial value ($16,000 vs. $12,500 for recipients), incur more debt in the form of loans from either financial institutions ($13,000 vs. $8,500 for recipients) or family and friends ($5,800 vs. $3,000 for recipients) and tend to obtain more funding from other unidentified sources ($26,400 vs. $12,600 for recipients).

As observed in a previous section (Section 2.1.3) analysis of debt data show that across both degree levels and areas of study, higher proportions of CGS recipients have no debt accumulated as compared applicants. For both doctoral and master’s recipients, the median debt was zero as compared to $4,000 for applicants. The median was also zero for SSH recipients but $8,500 for SSH applicants.

The average employment income from all sources is higher for CGS recipients ($75,000) than applicants ($68,000) (Table 14). It is also noted that employment income differs according to area of study.

Table 14: Annual Employment Related Income From All Sources, CGS Recipients and Applicants ($)

The analysis by cohorts indicates that the 2008-2011 cohorts ($60,287; standard error $1,595) earn less than the 2002-2007 cohorts ($80,357; standard error $1,266) which is to be expected given that, on average, they have been employed for less time.

2.3.2 International exposure

A slightly higher proportion of CGS recipients than applicants have had academic experiences as a student outside of Canada (for example, data collection, courses, field work, internships, language study or full degree program) both at the master's level (41.4% recipients vs. 37.2% applicants) and doctoral level (50.6% recipients vs. 45.5% applicants). Across areas of study, CGS recipients in NSE (47.5%) were more likely to indicate that they had had academic experiences as a student outside of Canada compared to NSE applicants (38.9%). However, CGS recipients (29.7%) and applicants (29.0%) in HS were the least likely to report academic experiences outside of Canada, compared to 46.0% of recipients vs. 46.7% of applicants in SSH. In the NSERC evaluation, 8% of the Postgraduate Scholarships (PGS) recipients at the master’s level vs. 11% at the doctoral level, reported participating in student exchanges with a university outside Canada and NSERC CGS-D recipients (21%) were statistically more likely to have had this international experience than PGS-D recipients.^{Footnote 36}

Respondents value their academic experiences outside Canada. Both CGS recipients and applicants generally agreed that international academic experiences were important to their graduate education and that international academic experiences were important to fulfilling their career path, more so at the doctoral level.

When asked how important it would be or would have been to gain international experience outside of Canada during their graduate degree if the opportunity was available, CGS recipients indicated that they most valued opportunities to participate in conferences/workshops (5.3 average), to enhance their future career opportunities (5.2 average), and to conduct collaborative research (5.0 average). CGS recipients also moderately valued opportunities to collect data/field work (4.7 average), and gain life experience (4.7 average). No significant differences were found across degree levels and areas of study.

Although CGS recipients moderately valued opportunities to gain some international experience, overall they did not value the opportunity to pursue a full graduate degree at an institution outside of Canada (2.6 average on a 7 point scale). The same was true of applicants (2.8 average). A similar conclusion is drawn from CGS recipients’ low interest in completing their degree abroad if the CGS scholarship had allowed it (average 3.1 on a 7 point scale).

Participation by CGS recipients in the CGS MSFSS is limited which is partly due to the design of the MSFSS program. The number of MSFSS is limited to a total 250 per year (which is equivalent to 5% of the 5,000 CGS awards available per year) and divided among the three agencies - CIHR (45), NSERC (80) and SSHRC (125). Among CGS recipients, the supplement appears to be more popular among doctoral students (9.4%) than master’s students (2.8%). Across areas of study, CGS recipients in HS were least likely to participate in the MSFSS (3.6%) vs. SSH (5.4%) and NSE (6.0%). CGS recipients who participate in the MSFSS are quite satisfied with their experience, including the opportunity to establish relationships and networks in other countries (6.5 average on a 7-point scale), to achieve outcomes they would not have achieved without the award (6.5 average), and to accomplish international research-related objectives (6.3). Recipients also expressed satisfaction with the duration of the award (5.9 average), but were less satisfied with the amount of the MSFSS (5.2). The MSFSS is restricted to research-related travel for 3-6 months in duration and requires a formalized host institution whereas travel fees to conferences/workshops and data collection/fieldwork are not eligible. Given the importance (average of 5.3 on a 7-point scale) that CGS recipients attach to opportunities to participate in international conferences/workshops, the ineligibility of travel fees to conferences/workshops could be seen as a barrier to the uptake of the MSFSS.

It should be noted that the lack of interest in pursuing full graduate degrees and other study opportunities outside Canada could be due to differences in the design characteristics of the CGS as compared to other available scholarships. Since CGS awards cannot be held outside Canada, students who want to pursue international studies would typically opt for an agency-specific scholarship - CIHR’s Doctoral Foreign Study Award (DFSA), NSERC’s Postgraduate Scholarship (PGS) or SSHRC’s Doctoral Fellowship (DF) - and therefore the low levels of interest observed in this evaluation could well be due to the possibility that those interested had already been excluded from the survey sample. Unfortunately, it is not possible to determine the number of students who did not apply for the CGS in the first place or declined it due to restrictions on taking up the scholarship abroad.

2.3.3 Comparisons with Corresponding Tri-agency Awards

Findings from this evaluation have been compared in various sections of the report with those reported in the SSHRC and NSERC scholarship and fellowship program evaluations as well as with the Vanier CGS Doctoral Fellowship program evaluation. Generally, similar trends were observable across the programs between the recipients and their corresponding cohorts of applicants. A direct comparison of the findings of the CGS and Vanier CGS Doctoral program evaluations along several indicators (Table 15) shows that each program can produce certain advantages. It is unclear however, what the causes of these differences are but since the Vanier CGS is newer, it is possible that its steady-state impact is yet to register.

Table 15: Relative Advantages of Vanier CGS and CGS Programs

In a similar broad comparison of the CGS Doctoral Fellowship and the SSHRC Doctoral Fellowship, what was striking was that there was no significant difference between the two types of awards across a large number of indicators (see Table C12, Appendix C).

2.4 Economy and Efficiency

2.4.1 Are the tri-agencies delivering the CGS program in a cost-efficient manner?

Several recent evaluations (e.g., Banting postdoctoral fellowships, 2015; Vanier CGS, 2014; and NSERC postdoctoral fellowships, 2013)^{Footnote 37} have approached program delivery efficiency from the perspective of ratio of operating expenditures to total award expenditure, total program expenditure, total number of applications or total number of awards. Depending on the denominator, the ratio would indicate the administrative cost for every dollar of fellowship awarded (usually expressed as a percentage), the administrative cost for every dollar of total program expenditure, the cost of administering one application or the cost of delivering one award. Data on the total direct costs to the tri-agencies of administering the CGS program from 2009-10 to 2013-14 are presented in Table 16. Administrative costs as a proportion of total program expenditures for the Vanier CGS program are presented for purposes of comparison.

The available evidence suggests that the tri-agencies are delivering the CGS program in a cost-efficient manner. The data indicate that the CGS program’s administrative costs as a proportion of its total program expenditures was 1.7% in 2009/10 and rose marginally to 1.9% over the last three years of the period covered by this evaluation. In comparison, the proportion for the Vanier CGS was 8.7% in 2009-10 and stabilized at 2.2% in the last two years covered by the evaluation. For the period from 2009-10 to 2013-14, CGS administrative costs as a ratio of total program expenditure was 1.8% as compared to 3.5% for the Vanier CGS program. The design and delivery of the CGS is quite different from that of the Vanier CGS program. It should also be noted that the CGS has a larger awards budget than the Vanier CGS ($748,198,109 vs. $98,094,237 for the comparison period) and this could have a dual effect. Firstly, the formula for calculating the efficiency ratios includes the awards budget as part of the denominator and therefore the difference could be partially explained as an artefact of the methodological approach used: larger budgets translate into lower ratios. Secondly, there is a fixed cost to setting up the basic administrative structures for running any program irrespective of the size of the award budget and beyond that, the larger the awards budget, the bigger the effects of economies of scale.

It is important to note that the expenditures discussed here do not include indirect and direct non-attributable costs, which could be substantial. Therefore, the expenditures are an underestimation of the total costs associated with the program and are computed this way to allow for comparability with the 2014 Vanier CGS evaluation results.

Table 16: CGS Program Expenditures by Year: 2009-10 to 2013-14

Universities incur direct and indirect costs in processing CGS applications and administering awards but it should be noted that this evaluation’s expenditure computations do not include those costs. Universities administer the CGS and other tri-agency awards using both staff and faculty resources. Faculties act as committee members to study applications and adjudicate scholarships to the most meritorious candidates while staff compile files and data, support meetings, document results and communicate with the tri-agencies.

While there are no definitive estimates of the costs incurred by universities, most institutions typically dedicate a part of an awards officer’s position to federal granting agency awards, plus faculty time promoting awards and sitting on review committees, some administrative time and some operating costs (communications, meetings). Two universities (one large, one small) estimated 40% of their awards office budget was used to administer federal granting agency awards but they could not separate out CGS costs specifically. Several administrators observed that their staff and faculty time spent on the CGS program has increased significantly with the harmonization of the CGS-M due to increased numbers of applications (students can now apply to up to five universities).

Stakeholder Perceptions of Program Delivery

Survey findings reveal that CGS recipients are satisfied with the monetary value of the award, but less so with the length of the award. Differences in satisfaction were apparent between recipients at the master's and doctoral levels. The latter, who receive a larger amount and can hold it for up to three years, were significantly more satisfied with both the value and the length of the award compared to recipients at the master’s level.

CGS recipients also reported significantly higher levels of satisfaction in regards to virtually all aspects of the application and selection process compared to applicants. Although recipients were more satisfied with the fairness of the selection process (5.7 average on a very dissatisfied (1) to very satisfied (7) scale) than applicants (3.4 average), both groups were least satisfied with the clarity of the selection process; disseminating more information on that aspect would be valuable. Furthermore, survey findings suggest that both CGS recipients (5.2 average) and applicants (4.2) feel that they are not promptly notified of the outcome of their application. The amount of work required to complete the application form was also identified as higher among applicants (4.1 average), unlike CGS recipients (5.5 average).

The key issue, however, revolved around the process used to evaluate CGS applications. Many recipients viewed the process as opaque and perhaps even somewhat arbitrary. A few recipients even spoke of having their application rejected in one year and then having “the same application” accepted the following year. The main issue that recipients wondered about, and this was shared by applicants too, was the criteria used to evaluate applications. In particular, both groups wanted to know more about how the application was scored and additionally for applicants, to receive feedback on which elements of the application were successful or unsuccessful:

• What weight is accorded to the supervisor’s CV?
• Are conference presentations worth as much as publications?
• How much do grades factor into the evaluation?
• What importance is attached to the reputation of universities?

Perceptions of Harmonization

Although the evaluation primarily covers the 2008-09 to 2012-13 period, to help determine the suitability of the current CGS program design and inform any decisions regarding changes to the program theory (i.e., links between program objectives and expected outcomes as currently stated in the logic model), the evaluation assessed key stakeholders’ perceptions of the harmonization process through key informant interviews with university administrators and CGS recipients’ supervisors and focus groups with current CGS recipients and applicants. Informants felt that harmonization has introduced some process improvements. There is now one application for the CGS-M which eliminates a previous risk of students applying to the wrong federal granting agency. Having one deadline for CGS-M for all three granting agencies is also an improvement from the past. It is helpful to be able to access applications online with a common password for all three federal granting agencies, and a common CV platform (the Canadian Common CV -CCV). It is important to note that the key informant interviews were conducted early in the data collection phase of this evaluation and after the first seven months of the implementation of the harmonization of the CGS-M. Over the course of the evaluation, the agencies have worked together and with the institutions to understand their concerns in relation to harmonization.

A few administrators felt that decentralization of CGS-M to universities under harmonization will negatively impact the reputation and effectiveness of the CGS as a national award. They reasoned that since universities will be making the award decisions, it will essentially be a university-level award that is funded by the federal government and suggested that, at a minimum, the federal granting agencies review applicants’ GPA scores for conformity with CGS requirements before universities undertake their reviews.

Most administrators and supervisors cited diverse operational challenges in administering the harmonized CGS-M. The consensus was that the harmonized process and the supporting technology were not fully developed with input from universities before this new approach was implemented in 2013. Other views expressed included a lack of clarity and openness about the communications from all three agencies regarding the program details and policies.

A few supervisors commented unfavourably about the 30% weight assigned to “research environment” and the use of number of publications as selection criteria while some administrators expressed mixed views about the allocation quotas under the harmonized CGS-M. A few universities have adapted the CGS criteria and application process for their own internal awards.

Most informants stressed that harmonization of the CGS-D should be delayed until the challenges with the processes for the CGS-M are fixed. University administrators feel that the CGS-D application process works reasonably well but noted specific challenges with decentralized payment processes (i.e., payments to departments vs central university) used by one of the three agencies. This suggests the need to harmonize the payment process across the three agencies in future as the current varying processes make it difficult for the institutions to manage due to communications and administrative problems - and inefficiencies - for universities through decentralized payments. Some administrators cited other problems such as lack of information on who to contact for information or inability to reach that person, vagueness of rules (e.g. one agency provides six months of parental leave versus four months for the two other agencies).

3.0 Relevance

3.1 Continued need for the CGS program

The continued need for the CGS program was assessed in terms of trends in the demand and supply of highly qualified personnel (HQP) in Canada, perceptions of key stakeholders as to whether the program was relevant and the proportion of successful applicants who declined the award.

The CGS program seeks to ensure a reliable supply of HQP to meet the needs of Canada's knowledge economy. The strategy to achieve this objective is to provide incentives for increased enrolment in graduate studies in Canada; that is, by awarding scholarships to a larger number of qualifying students and making those awards financially attractive.

The CGS program's objectives were designed to support the Government's goal of making Canada one of the most innovative countries in the world by moving from 14th place to among the top five in R & D investment per capita in the OECD, an objective which is yet to be achieved^{Footnote 38}. At program inception, it was estimated that to conduct the amount of research needed to rank in the top five, Canada needed an additional 100,000 highly qualified R & D employees, of whom a significant proportion must be researchers with advanced degrees.^{Footnote 39} To help create those researchers, the then Government's Innovation Strategy document - Achieving Excellence^{Footnote 40} – set the target of a 5% increase per year in graduate student enrolment at Canadian universities and it identified doubling the number of federal government master’s and doctoral scholarships as a priority for achieving that target.

On the supply side the available evidence speaks to an increase in graduate enrollment in Canadian universities since the inception of the CGS program. An Association of Universities and Colleges in Canada (AUCC) report noted that the number of students in full-time master’s and doctoral programs increased from 71,000 in 2000 to 127,000 in 2010 averaging a growth rate of 8% per annum.^{Footnote 41} Statistics Canada data also indicated an increase in graduate enrollment from 175,000 in 2008-09 to 208,000 in 2013-14.^{Footnote 42} On the demand side, data on postgraduate employment is difficult to obtain; the most recent being Statistics Canada’s Survey of Earned Doctorates (SED) for which data was only collected between 2003-04 and 2007-08 and the evidence confirms a growing demand for knowledge workers in Canada’s labour market. In an analysis of the SED data, the AUCC estimated that the number of jobs filled by university graduates more than doubled from 1.9 million in 1990 to 4.4 million in 2010.^{Footnote 43} Additionally, Human Resources and Skills Development Canada projected that 1.4 million new jobs would be created between 2008 and 2017 and that 75% of these would require postsecondary education.

In relation to the higher education sector specifically, the AUCC (2007) projected that in the decade after 2007, the number of new faculty members would grow from about 40,800 full timers to between 9% and 33% more and an additional 21,000 positions would need to be replaced due to retirement or other reasons.^{Footnote 44} Although somewhat dated, the data would seem to confirm a continued and persistent need for HQP in Canada’s higher education sector.

Additional evidence shows that HQP are being produced and are finding employment in the fast growing occupations - health professions, engineering and a variety of business and management occupations^{Footnote 45} – thus helping Canada to compete in the new global knowledge economy.

The perceptions of key stakeholders were also considered in assessing the need for the CGS. Key stakeholders felt that Canada could not compete globally in the knowledge economy without HQP and that to enable students go through the required advanced training, the federal government should help with funding, particularly for those who lacked the means.

All key informants felt that the CGS award is needed as part of a larger suite of other training awards. The program niche is that it focuses on high calibre students who want to pursue research and provides sufficient funding to allow them to do this. This helps develop their career path in their field of research and contributes to the federal government’s objectives for innovation and research excellence. Some felt that without CGS and other awards programs there would be a serious gap in research in Canada.

It was noted that other awards of the three federal granting agencies and provincial awards also fund research excellence but usually at a lower funding level. Top students get the CGS awards and other high calibre students may get other federal granting agency or provincial/institutional awards or university funding. This array of awards was considered effective as it allows the top students to focus on their research while supporting others with a lesser interest in pure research.

Most key informants did not see any overlap between the CGS and other federal granting agency awards including the Vanier CGS. No areas were identified where the CGS and the Vanier awards worked at cross purposes yet some informants felt that in spite of the difference in prestige associated with the higher monetary value of the Vanier, there was considerable overlap in the two awards in terms of fostering research excellence.

The proportion of successful applicants who decline an award can indicate whether there is a continued need for the award or better alternatives exist. In the case of the CGS, findings from the administrative data analysis showed that successful applicants made use of the award and only a few declined it. Their weighted proportion among survey respondents was less than 5%.Of this 5% who declined, the most common reasons were being accepted into a study program outside Canada (42%), remained in Canada but got accepted in other degree programs for which they could not receive the CGS award that they applied for (19%),decided to join the workforce (14%), decided not to pursue the degree program, had timing issues, did not successfully gain admission into a program, or for personal or other reasons (17%). Only 8% of the 5% who declined the award did so because they received funding that was more generous than the CGS or that could be held for a longer period.

3.2 Consistency of CGS program with federal roles and responsibilities

Recent Government of Canada reports and publications such as Seizing Canada’s Moment: Moving Forward in Science, Technology and Innovation Report^{Footnote 46} affirm the federal government’s role in supporting postgraduate education to develop, attract and retain highly qualified personnel (HQP) who can contribute to Canada’s growth as an innovative, knowledge based economy. By aiming to “help ensure a reliable supply of highly qualified personnel to meet the needs of Canada’s knowledge economy (CGS Terms and Conditions),” the CGS program aligns with federal roles and responsibilities and specifically with the mandates of the three federal research funding agencies as outlined in their respective enabling legislations.

The CIHR mandate is to: “foster the development and ongoing support of the scientific careers of women and men in health research;” (CIHR Act, Section 5(b) and to: “provide funding to promote, assist and undertake health research and to otherwise carry out its objective” (CIHR Act, Section 26(a)).

Similarly, NSERC was established to: “promote and assist research in the natural sciences and engineering, other than the health sciences;” (NSERC Act Sections 4(1)(a)) and “expend, for the purposes of this Act, any money appropriated by Parliament for the work of the Council or received by the Council through the conduct of its operations” (NSERC Act Section 4(2)(a)).

In the same vein, the SSHRC is authorized to: “promote and assist research and scholarship in the social sciences and humanities;” (SSHRC Act, Sections 4(1)(a)) and “expend, for the purposes of this Act, any money appropriated by Parliament for the work of the Council or received by the Council through the conduct of its operations;” (SSHRC Act, Sections 4(2)(a)).

The results of the qualitative and quantitative data analysis were consistent with these document review findings. Participants in the key informant interviews considered the federal support for graduate training to be in line with federal roles and responsibilities in ensuring that Canada was competitive globally. They felt that a highly educated workforce with high human capital is a public good and thus the federal government has a role in ensuring that this is achieved.

“Canadian students need to be able to compete globally.”

Additionally, survey results show that the CGS program supports students financially during their studies, allowing them to better develop their skills (see Section 2.2.2 of this report). The reach of the CGS, that is the number of recipients it funds annually, suggests that it is filling a critical need for funding support at the Master’s and Doctoral levels.

3.3 Alignment of CGS program with Government of Canada priorities

The CGS program has remained a priority for the federal government since it was launched in 2003. Additionally, the program’s objectives align with the strategic priorities of the tri-agencies to build research capacity through attracting, developing and retaining research talent.

When the CGS was announced in Budget 2003, it was provided with funding that reached an annual amount of $105 million when fully phased in by 2006-2007. Budget 2007 increased the annual budget, when fully ramped up, to $131.25 million and Budget 2009 provided additional temporary funding of $87.5 million over three years, starting in 2009-2010 (Renewal of CGS Terms and Conditions, p.2). Beyond these budgetary allocations, government policies have also directly and indirectly referenced and supported the CGS program. The current federal government policy on science, technology and innovation notes that the CGS encourages “Canadians to pursue advanced education and conduct research” and “enables students, across all disciplines, to develop the skills needed to become future research leaders and highly-qualified personnel across all sectors of the economy.”^{Footnote 47}

Also, the CGS maps directly onto the corporate mandates and strategic priorities of the tri-agencies. CIHR seeks to build the capacity of the Canadian health research community through the development of researchers and the provision of sustained support for scientific careers in health research. Further, CGS is supported in CIHR’s Roadmap under the strategic area “Invest in world-class research excellence” through attracting and retaining the best international scholars and experts. In 2009, CIHR pledged to support the training, retaining and sustaining of a healthy research foundation by training, attracting and retaining the best talent in health research. Additionally, in its updated strategic plan, Roadmap II, under “Strategic Direction 1: Promoting Excellence, Creativity and Breadth in Health Research and Knowledge Translation” and Section “1.2: Building a Solid Foundation for the Future,” CIHR pledges its support for training and mentoring the next generation of researchers and CGS represents one of the programs supported by CIHR to address this strategic area.^{Footnote 48}

Similarly, NSERC aims to develop the next generation of talented scientists and engineers through its scholarships and research stipends. NSERC contributes to building a stronger culture of science, technology and innovation in Canada by supporting the attraction, retention and development of highly qualified people in the natural sciences and engineering in Canada through various programs, including scholarships and stipends. This priority is highlighted in the 2016-17 Report on Plans and Priorities in its People - Research Talent Program which falls under the Strategic Outcome “Canada is a world leader in advancing, connecting and applying new knowledge in the natural sciences and engineering.”^{Footnote 49}

SSHRC prioritizes investments in developing an emerging generation of top researchers in the social sciences and humanities through their primary strategic outcome “To make Canada a world leader in social sciences and humanities research and research training” (SSHRC, 2013). Research training is promoted under the umbrella program of Talent, and as outlined in their new Strategic Plan 2013-2016, SSHRC continues to support the attraction and retention of research talent and enhance the quality of research training through the provision of targeted programming.

4.0 Conclusions and recommendations

4.1 Performance

4.1.1 Immediate outcomes

The extent to which the CGS program is achieving its immediate outcomes is mixed. In relation to its intended role as an incentive for enrolment in graduate studies, the evaluation finds that the self-reported primary motivation for students to pursue a graduate degree remains a deep interest in the area of study. This is consistent with the finding that over four-fifths of the students were already enrolled before obtaining their CGS or state that they would have enrolled regardless, whereas only about one in ten (13.2%) would not have enrolled in a program had they not received a CGS award.

Graduate enrolment in Canada ranged from ~160,000 in 2007-08 to ~190,000 in 2012-13. The CGS target of funding 5,000 scholarships a year cannot, on its own, increase enrollment numbers significantly. That said, the CGS has made a clear contribution to the ability of students to devote more time to their studies. CGS recipients abandon their studies less frequently; they accumulate less debt; and completed their degrees faster. Nevertheless, the total duration of their studies remains strongly affected by concurrent factors such as the nature of the research process itself.

University professors, administrators and graduate students are well aware of the CGS and see it as part of the suite of federal training awards.

4.1.2 Intermediate outcomes

The CGS is achieving several of its intermediate outcomes. An aspect of the logic of the program is that it provides a means of living while studying full-time, allowing students to devote more time to their studies. Survey findings provide consistent evidence to support this logic: a larger proportion of CGS recipients than applicants completed their degree within the time frame of eligibility for the survey (i.e., competitions launched 2002-2011); and, among those who have completed their degree, recipients also completed faster than applicants.

The amount of the CGS award is also seen as affording opportunities that enrich the research training experience for recipients such as conference attendance for example. Survey data confirm that recipients are more involved in research-related activities and are more productive in terms of publications and communications than applicants.

The evaluation found that CGS recipients and applicants were satisfied with opportunities to develop their research skills and personal/professional skills, with recipients more satisfied than applicants. In addition, CGS is perceived as helping attract highly qualified researchers to universities which create a synergy whereby universities with highly qualified researchers are, in turn, attractive to top graduate students, including CGS recipients.

Graduates find employment related to their studies, with CGS recipients performing better than applicants. Recipients also reported that personal/professional experience and research-related experience helped them in obtaining the position they currently hold. A majority of doctoral graduates are largely employed in university settings whereas the majority of master’s students work in the private sector or with government. This evidence indicates that the CGS program has contributed to increasing the capacity to meet demand for highly qualified personnel (HQP) in the faculties of Canadian universities and in the public and private sectors.

The CGS scholarship is recognized as distinctive by professors, graduate students and administrators in Canada and promote Canadian research excellence abroad through their support for increased productivity in terms of publications and communications worldwide. However, the multiplicity of scholarship names within the CGS program may hinder brand recognition.

4.1.3 Long-term outcomes

The CGS program has contributed to long-term impacts on the career paths of participants, generally validating recipients’ decision to pursue a research career, facilitating the time spent studying and perfecting research and other skills. This is demonstrated by the findings that CGS recipients produce a greater number of articles and presentations, are more likely to be currently employed in a position closely related to their degree program and earn more than applicants.

Overall, CGS recipients have slightly more international exposure than applicants thus extending Canada's reputation abroad. While CGS recipients moderately valued opportunities to gain some international experience, overall they did not value the opportunity to pursue a full graduate degree at an institution outside of Canada and the same was true of applicants. In addition, recipients had little interest in completing their degree abroad if the CGS scholarship had allowed it. It should be noted though that since CGS awards cannot be held outside Canada, students who want to pursue international studies would typically opt for an agency-specific scholarship - CIHR’s Doctoral Foreign Study Award (DFSA), NSERC’s Postgraduate Scholarship (PGS) or SSHRC’s Doctoral Fellowship (DF) - which may explain the low levels of interest observed in this evaluation. CGS recipients can obtain some international experience though the CGS Michael Smith Foreign Study Supplement (MSFSS) which is a one-time award of up to $6,000 to undertake 3-6 months of study outside of Canada. Participation in the CGS-MSFSS is restricted to 250 awards per year and recipients are quite satisfied with the experience.

4.2 Economy and efficiency

Available evidence suggests that the tri-agencies are delivering the CGS program in a cost-efficient manner. For the period 2009-10 to 2013-14, administrative expenditure (direct attributable costs only) as a proportion of total expenditure (direct administrative costs and award expenditure) ranged from 1.7% to 1.9%. The average for the period is 1.8% as compared to 3.5% for the Vanier CGS program. The difference in ratios could be partially due to the effects of economies of scale given that there is a fixed cost to setting up the basic administrative structures for running any program irrespective of the size of the award budget and that the CGS’ award budget for the period is about eight times that of the Vanier. The costs do not include the resources that universities devote to administering the CGS program. It should be noted that the expenditures presented in the analyses do not include indirect and direct non-attributable costs which could be substantial. Therefore, the expenditures are an underestimation of the total costs associated with the program and are computed this way to allow for comparability with the 2014 Vanier CGS evaluation results.

Recipients appreciate the value of the scholarships but question its duration in relation to actual time to degree completion while applicants perceive the adjudication process as opaque and possibly unfair and ask for information about how applications are scored and which elements are successful or unsuccessful. The centralization brought about by the CGS-M harmonization process has improved deadlines, the application process, and the distribution among agencies according to areas of expertise. However there were some reservations about the availability and clarity of the information from the three agencies, and about the full readiness of the supporting technology and, for one agency, about the efficiency of making payments directly to university faculties or departments instead of to the central university awards administration.

4.3 Relevance

The evaluation confirms the continued need for the CGS program to foster excellence in graduate studies and research through financial support, enabling high achieving students to pursue graduate degrees regardless of their financial means and to devote more time to their studies, thus being more productive. There is however, mixed evidence for the extent to which the program is meeting some of its immediate objectives (e.g., CGS as an incentive to enroll in graduate studies) in that it cannot be demonstrated that the objectives are being met. At the same time, there is evidence that up to 13% of CGS recipients would not have entered a graduate program without the scholarship; that the recipients worry less about their financial situation than other applicants; and that recipients complete their degree more often, in less time, spent more time on research training, and produced more publications and communications.

The CGS program aligns with federal roles and responsibilities and the mandates of the tri-agencies to develop HQP who can contribute to the growth of Canada’s knowledge economy as outlined in the 2014 Science, Technology and Innovation Strategy. Additionally, the program is consistent with federal government and tri-agency priorities.

4.4 Recommendations

The evaluation evidence confirms the effectiveness and continued relevance of the CGS program and is needed to support HQP development to insure that Canada’s knowledge economy remains globally competitive well into the future. While evidence to support the achievement of some of its immediate outcomes is mixed, the program is achieving several of its intermediate and longer-term outcomes. The program is being run efficiently. The following recommendations are made to further enhance the program’s ability to deliver on its mandate in the light of the current evolving context of program changes across the tri-agencies and program management’s actions taken in response to the 2008 CGS evaluation’s recommendations.

1. Review and revise the Canada Graduate Scholarship program’s expected outcomes and strengthen performance measurement.

Since the introduction of the CGS in 2003 and the 2008 evaluation, the graduate level training landscape in Canada has undergone substantial change. Newer programs - the Vanier-CGS and the Banting postdoctoral fellowship programs - have been introduced to attract and retain the best doctoral and postdoctoral level trainees respectively and the objectives of these programs have created, by de facto, a new hierarchy across the suite of federally-funded training awards.

The evaluation found evidence of the program’s relevance and the broad need for the CGS, though there is mixed evidence for the extent to which the program is meeting its specific objectives. In particular, the evaluation found that the CGS program has limited ability to increase incentives for, or enrollment in, graduate studies. This finding calls into question the logic underlying the program as it cannot be demonstrated that two of the four immediate outcomes of the program are being achieved (the objectives to (a) increase incentives to enrol and (b) increase enrollment) and is consistent with the 2008 evaluation recommendation to rethink the program logic.

The evaluation could have benefited from good quality performance data but a major drawback is that there is currently no end of award reporting tool although there are plans to prepare one. In response partly to the 2008 evaluation recommendations and partly due to the ongoing harmonization, the program’s objectives and their hypothesized links to intended outcomes are being revised and updated. As these revisions are implemented and to be consistent with the Vanier-CGS and the Banting postdoctoral fellowship programs, a performance measurement strategy and end of award reporting tool should be adopted to enable ongoing tracking of recipients.

2. In the context of Canada Graduate Scholarship harmonization across the Tri-Agencies, the program should provide more information on the review process and outcomes to applicants, and also explore opportunities for branding the program under a single name.

Plans for harmonizing the CGS-D are proceeding and preliminary evaluation findings have informed aspects of the process including validating the newly proposed core principles for the CGS-D and clarifying program objectives.

In the context of harmonization and the increased role for institutions in a harmonized program, the CGS program should work to improve transparency of review processes and feedback to applicants. Although the evaluation found, perhaps not surprisingly, that recipients were more satisfied with the fairness of the selection process than applicants, both groups found the selection process unclear. This suggests that more information on the application process would be valuable, as would prompt notification of the outcome of their application. The transparency of the review process was thought to be an issue. Reviewers and, post-harmonization, institutions should be encouraged to provide more detailed feedback to applicants in order to build capacity for future applications or to build understanding for reasons for why they were or were not funded.

The evaluation found that the CGS program is recognized as distinctive by university professors, graduate students and administrators in Canada however, the multiplicity of scholarship names within the program may hinder brand recognition.

Participants in the key informant interviews suggested simplifying the naming of the CGS awards and this was supported by focus group participants very few of whom referred to the CGS as the "Bombardier", "Bell" or "Banting-Best" Scholarship although there was some degree of recall of these names. Exploring opportunities to rebrand the CGS under a single name might thus be warranted.

Appendix A: CGS Program Logic Model

Appendix B: Methodological Details

B.1 Key Informant Interviews

A total of 25 interviews were planned with four respondent types including university graduate award administrators (n=9), supervisors of CGS recipients (n=6) and employers who hire master’s and doctoral graduates (n=5), and senior CGS program management (n=5). The final numbers of interviews are reported below for each group. Interviews with the first three respondent groups were conducted by a private contractor, Goss Gilroy Inc. (GGI) while the last group was conducted by the CIHR Evaluation Unit.

B.1.1 Identification and selection of key informants

Administrators: CIHR provided a list of 15 potential key informants with contact information from universities across Canada. University administrators were sampled using a purposive approach that stratified the population of Canadian universities who have a CGS and Vanier CGS award allocation by Federal Granting Agency, geographic location (Western, Prairies, Central, Atlantic) and by university enrolment rates (Small, Medium, Large). Care was also taken to not overburden interviewees who had participated in the Vanier evaluation conducted a few months earlier or were expected to participate in the soon to commence evaluations of Agency Specific Awards programs. The consultant made contact with all possible participants in order to reach the target of nine completed interviews.

Supervisors: CIHR provided a list of supervisors of CGS award winners from each of the three granting Agencies (17 in total). The population of current supervisors of CGS recipients were stratified based on Federal Granting Agency, university enrollment rates (Small, Medium, Large), and degree type of student supervised (master’s or doctoral). Interviewees from within population sub-groups were then selected based on a random selection of current CGS recipients, in the final year of their award. The consultants attempted to contact all possible participants.

Employers: CIHR provided a list of 16 private and public sector organizations that were considered to be potential employers of CGS award winners from across the three federal granting agencies. A senior representative of HRSDC talent and recruitment was selected to provide their perspective on public sector employment of graduate students. Organizations in the private sector were selected based on Canada's Top 100 Corporate R&D Spenders List and on recommendations by members of the Evaluation Working Group and were stratified across the industries of potential employment of the three Funding Agencies (Health Sciences, Social Sciences and Natural Sciences and Engineering). The consultant then did a search through telephone calls and the internet to identify potential key informants in each organization in senior roles in research or human resources management. A total of 14 potential key informants were identified and attempts were made to contact them.

Senior program management: At each of the tri-agencies, the director-general most closely involved with the CGS program was identified for interviewing along with staff of the Tri-Agency Harmonization Committee making a total of five potential interviewees. This group of interviews was conducted by the CIHR Evaluation Unit.

All interviews were conducted by telephone except for those with the tri-agency CGS program management which were in-person. The interviews used a semi-structured interview guide tailored for each key informant group. Questions in the guides were developed based on the evaluation issues to be addressed through the interviews but worded to be appropriate for interviews. The guide was pre-tested with the first interviewees in each group. Interviews were conducted by telephone in either English or French, depending on the preference of the respondent. Interviews lasted between 45 to 60 minutes. All interviews were audio recorded with the consent of the key informant.

Table B1: Key Informant Invitation Results

B.1.2 Analysis

The notes for each key informant were compiled into a matrix by interview question. The interview notes were then organized in a second matrix by key informant group according to each evaluation question. Data relevant to each evaluation question were reviewed to identify common themes and any differences in opinions across the key informant groups. Based on this analysis, overall findings were developed for each evaluation question, along with the supporting evidence according to key informant group, and noting any corroboration or differences in opinions across key informant groups.

B.1.3 Limitations

There were a small number of key informants overall and within each key informant group, which made it difficult to form overall findings and draw conclusions. In particular, there were only three employers interviewed and therefore all direct references to interview findings with the employer category were deleted from the main body of this report.

B.2 Focus Groups

Focus groups were used to gather qualitative, experiential data from current/recent CGS award recipients and from students who applied for a CGS award and did not receive one, but who may have received other financial support for their studies (e.g. from their school, supervisor, another award including those from tri-agencies, loans or family).

Two in-person focus groups were conducted at each of three locations across Canada. At each location, a focus group was held with CGS recipients funded through a federal granting agency (Ottawa - CIHR; Montreal - SSHRC; Vancouver - NSERC); and a counterfactual focus group with unfunded CGS applicants who applied through a federal granting agency. The focus groups in Ottawa and Vancouver were conducted in English and the Montreal focus groups were conducted in French.

Participation in each focus group was targeted to include approximately eight participants. To engage this number, the recruitment attempted to engage 12-15 participants for each group. Efforts were made to enroll a balance of master’s and doctoral level recipients/applicants in each group.

B.2.1 Identification and Selection of Participants

CIHR provided lists of potential participants for each focus group. The consultants sent an email with an introductory letter from CIHR (in both official languages), and followed up by telephone. Confirmed participants were sent a reminder email the day before the focus group. Up to three contacts were made with each individual. Each was offered a $50 honorarium for their participation. Table B2 shows the results of this process.

Table B2: Focus Group Invitation Results

B.2.2 Analysis

The notes for each focus group were compiled into a matrix by focus group question linked to the related evaluation questions. The notes were then organized by recipient/applicant groups for each evaluation question, noting response of master’s and doctoral level students. Data relevant to each evaluation question were then reviewed to identify trends, themes and any differences between recipients and applicants as well as master’s and doctoral level students.

Based on this analysis, overall findings were developed for each evaluation question, along with the supporting evidence according to CGS recipients/applicants, master’s and doctoral level students, and noting any corroboration or differences in opinions across these groups.

B.2.3 Limitations

The contact information was not current for many of the individuals in the sample. While the intent was to identify a random sample at each location, it is not clear how representative the participants are of previous CGS recipients and applicants. Those who have completed studies or moved elsewhere are likely under-represented.

B.3 Document Review

Based on a review of the evaluation framework, the following key areas were addressed in the literature review:

• Identified role for the federal government in supporting research graduate students in Canada
• Compatibility of CGS program objectives/expected results with federal granting agencies’ and federal government’s priorities
• Change in application pressure to Canadian universities for graduate studies
• Change in graduate enrollment rates at Canadian universities
• Completion time of graduate students
• Trends in graduate enrollment rates for graduate studies at Canadian universities by discipline
• National data trends in HQP and comparisons to other countries where possible
• Extent of duplication/overlap of CGS program objectives with other federal, provincial and institutional graduate funding opportunities.

B.3.1 Identification of key documents

Following the identification of key areas of interest, a search was conducted to identify key documents.

Evaluation reports

• An evaluation of CGS was completed in 2008. Various technical reports were reviewed to inform the methodology of this literature review.
• The completed Vanier-CGS Evaluation Report was consulted to inform comparison data on national and international programs.

Government documents and websites

• Evaluation Design Report;
• Agency-specific websites and strategic plan documentation; and
• Federal government documentation, such as the S&T Strategy and Budget 2014.

Statistics Canada

• Various Statistics Canada tools and reports were used to provide information on enrolment and HQP supply and demand

Organizational Websites

• Various websites of organizations that concern themselves with graduate student training in Canada were consulted, including the Council of Ministers of Education, Council of Canadian Academies, Canadian Council on Learning, Association of Universities and Colleges of Canada, the Canadian Association for Graduate Studies, the Organisation for Economic Co-operation and Development,
• Various Canadian university websites

Journal Articles

• Online journal databases, such as JSTOR were accessed to support findings regarding identified best practices of research training. The search was conducted using key words, such as “graduate student” + funding.

Internet Search

• Google was used to identify any additional organizations and relevant materials. Key words used in the journal database search were used.

B.3.2 Analysis

All data collected through the above methods were entered into NVivo for data analysis. In the case where electronic versions of information were unavailable, detailed notes were created and imported into the NVivo file.

Data were coded according to the evaluation questions and indicators detailed in the Evaluation Matrix. Findings were then synthesized into a report following the same structure.

B.3.3 Limitations

Comparator data between CGS recipients and non-recipients was not included in the document review technical report. The literature provided little information; however, data from the survey were used to make the relevant comparisons.

In addition, a key limitation for this review is that the literature and reporting from Statistics Canada view disciplines differently. Therefore, it was not possible to present data about disciplines that aligned completely with federal tri-agency disciplines in this report.

B.4 Surveys

B.4.1 Survey development and implementation

A pretest was conducted online in English with a sample of CGS recipients receiving awards during two time periods (2002 to 2007 and 2008 to 2011), as well as those who were awarded the CGS but declined the award. The pretest was conducted between March 24th and 28th, 2014 with an initial invitation sent on March 24th and a reminder to non-respondents sent on March 26th. Those invited were told that the deadline for response was March 27th. During the pretest phase, the survey was completed by 13 recipients from the 2002-2007 cohort, 18 recipients from the 2008-2011 cohort and two of those who declined the award. Advance notice from CIHR was provided to survey respondents via email. The survey was conducted online. Survey invitations were sent on July 29th, 30th and 31st, 2014 to 9,774 award recipients from 2002 to 2007, 13,178 award recipients from 2008 to 2011 and 1,906 recipients who declined the award. Three reminders were sent with cases being collected up to October 21st. The average length of the survey was 24 minutes; seven minutes for those who declined the CGS award, 13 minutes for award recipients from 2002 to 2007 and 34 minutes for award recipients from 2008 to 2011.

Similarly, a pretest was conducted online with a sample of federal granting agency scholarship applicants during two time periods (2002 to 2007 and 2008 to 2011), in both English and French. The pretest was conducted between October 21st and 31st, 2014 with an initial invitation sent on October 21st and a reminder to non-respondents sent on October 28th. During the pretest phase, the survey was completed by nine 2002-2007 scholarship applicants and 28 from the 2008-2011 cohort. Advance notice from CIHR was provided to survey respondents via email. The survey was conducted online and invitations were sent on November 12th, 2014 to 16,172 scholarship applicants in the 2002-2007 cohort, and 14,960 in the 2008-2011 cohort. Four reminders were sent and cases were collected up to January 30th, 2015. The average length of the survey was 18 minutes; 10 minutes for 2002-2007 scholarship applicants and 23 minutes for 2008-2011 scholarship applicants.

The response rate for the valid sample of Recipients was 23% (Groups A & B) and 12% for Applicants (Groups D & E) (Table B3).

Table B3: Sample Sizes and Response Rates for CGS Recipients and Applicants Surveys

B.4.2 Statistical analyses

All statistical analyses were performed using IBM’s Statistical Package for the Social Sciences (SPSS).

The primary goals of the survey data analyses were to provide a description of the participants’ profile in terms of sample size and background characteristics and to compare the results from CGS recipients against those of applicants for each indicator in each outcome.

Profile analysis

A profile analysis was initially conducted to provide descriptive information on sample size (N) and proportions (%) of respondents broken down by the following categories:

• Group membership (scholarship recipient, decliner, applicant) and subgroups (2002-2007 and 2008-2011 cohorts)
• Degree level (master’s, doctoral)
• Areas and sub-areas of study within Social Sciences and Humanities, Natural Sciences and Engineering, and Health Sciences
• Region, size and type of institution of provenance
• Gender
• Age at application time
• Minority status
• Presence of dependents
• Undergraduate grade point average
• Preferred language.

The results of the profile analysis are not presented directly in this evaluation report but are rather used in the background to facilitate other aspects of the analyses such as the weighting of samples.

Missing values

For the profile, categories of data such as “not applicable”, “don’t know” or absence of response were reported as such. For the inferential statistics comparing recipients versus applicants etc., those data were coded as missing and excluded from the analyses.

Weighting

A preliminary analysis compared the distribution of respondents in various categories of administrative data provided by the agencies against the population distribution for recipients (including decliners) and applicants separately. The available categories were:

Table B4: Administrative Categories Available to Compare Sample and Population Distributions

Some initial disparities were found exceeding a threshold of 3%. Weights were developed to correct for those and applied to all analyses. The inferential statistics were conducted with the weighted data exclusively.

Table B5: Initial Disparities Above 3% Threshold Between Sample and Population Distributions

Inferential Statistics

While a primary goal is to test for significant differences between recipients and applicants in relation to the indicators for each outcome, there is also a need to separate these tests by degree level (master’s or doctoral) and area of study (SSH, NSE, and HS) because certain aspects of the graduate student experience are modulated by these additional factors. Accordingly, most analyses initially present tests of differences between recipients and applicants overall^{Footnote 50}, within degree level, and within area of study. In addition, wherever feasible, tests of main effects of degree level (i.e. master’s vs. doctoral, pooling recipients and applicants) and area of study (i.e. SSH vs. NSE vs. HS, pooling recipients and applicants), are also presented in the tables.

Frequencies and Percentages vs. Means

Some survey questions collect the frequency of occurrence of discrete events. For example, yes/no answers to questions such as:

“According to our records, you accepted a Canada Graduate Scholarship (CGS) to study at the [DEGREE TYPE FROM RECORDS] level. Is this correct?” or

“Which one of the following best describes the field of your [DEGREE TYPE FROM RECORDS] studies?

• Social sciences and Humanities
• Natural Sciences and Engineering
• Health Sciences.”

For these questions, the results are cross-tabulated and differences between percentages of recipients and applicants are tested for statistical significance using the z test (similar to Chi-square) at an alpha level of 0.05 for the Type I error rate. A Bonferroni adjustment is used when more than one significance test is performed within a dimension of the plan to maintain the overall risk of falsely declaring significance at no more than 5%. For example, since there are three areas of study, there are three comparisons between recipients and applicants to be made: one within each area (SSH, NSE, and HS). Variations in aspects of the graduate student experience by degree level or area of study add another dimension of complexity to the multiple comparisons. For example, teaching occurs more frequently during doctoral studies than during master’s and laboratory work occurs less frequently in humanities than science. At the same time, the sample contains more doctoral students than master’s, more humanities than science or health and only a few applicants participated in the survey at the master’s level and any statement about recipients vs. applicants must take all these variations into account simultaneously. With the cross-tabulations, the overall applicant vs. recipient test aggregates degree levels and areas of study; the applicant vs. recipient within degree level still aggregates over area of studies, etc. and the effects can be distorted. For example, it is possible that in some percentage tables, a trend observed in sub-groups could be reversed at the aggregate level when comparing recipients and applicants due to the much larger representation of recipients among master’s students than among doctoral students (particularly in SSH and HS). To mitigate this distortion the percentage tables in this report present recipient vs. applicant comparisons only within degree level and/or area of study but not for the “total” or “overall” sample. The problem is adequately addressed in the case of ratio measures as explained below.

In the recipient and applicant surveys, many questions collect answers on a scale ranging from 1 to 7 and sometimes from 1 to 5. The scales capture notions of Improvement, Extent, Frequency, Satisfaction, Relation, Prestige, Impact, Agreement and Usefulness. The points on the scales are labeled with adjectives describing amount or intensity. Some scales, such as Satisfaction, Impact and Agreement are conceptually bipolar yet formulated to yield positive integer values. For example, Satisfaction ranges from (1) “Very dissatisfied” to “Very satisfied” (7) with “Neither satisfied nor dissatisfied” (4) as the midpoint.

The other scales (Improvement, Extent, Frequency, Relation, Prestige, Impact and Usefulness) are unipolar beginning with labels such as Not at all (1) for Extent or Very low (1) for Prestige for example. There are two possible strategies to analyze such ordinal scales. They can be treated as categorical counts, cross-tabulated and tested with a z test as outlined above. Another strategy is to treat the data as near-continuous. This is justifiable because even though there are only a few (7) scale points on the text of the question, the underlying attributes represented by the scale are clearly continuous. One can easily imagine a degree of Improvement or Satisfaction or Prestige falling between two other amounts.

The advantage of this strategy is to allow the application of Ordinary Least Squares (OLS) techniques which, in turn, enable the analysis to incorporate all the dimensions of the analysis simultaneously. OLS do this very well by computing estimated marginal means derived from regression equations.

The use of analysis of covariance (ANCOVA) eliminates the correlation effect due to unequal sample sizes^{Footnote 51}. In addition, in spite of the smaller sample size, the Standard Error of the Mean (SEM) was not much larger in HS than in the other areas.

Cross-tabulations would require dichotomizing the seven points scales yet it is not clear where the correct cut point is. On the central question of Prestige for example, [Very low (1), Low (2) and Moderate (3) High (4) and Very High (5)], should the midpoint be grouped with prestige or no prestige? This is important not just because of the meaning of the words but also because the location of the cut may actually determine the significance of a difference between groups. Another risk arising from dichotomizing is that the techniques to test for significant differences are less elaborate and force a large number of simple comparisons where the control for Type I error becomes so stringent that the test loses its power.

In addition to its ability to more accurately evaluate the applicant vs. recipient effect by simultaneously adjusting for other design factors such as degree level and area of study, the OLS enable further statistical adjustments for other extraneous variables such as gender, age, minority status or presence of dependents by including them as covariates in the mathematical model. A preliminary examination indicated that there were too many missing answers to the questions about age, minority status or presence of dependents. Their inclusion in the model would have caused the loss of at least 500 observations. But gender was retained.

For all their advantages, OLS techniques also have their limitations. They require that the dependent variable be normally distributed and, even more crucially, that the residual error variance in the cells of the research design be homogeneous. When these assumptions are not met, the p value of the significance test is inflated above 0.05. Verifications (Levene’s test) for each question of the survey that was eligible to OLS analysis indicated that the residual error variance was heterogeneous in many cases. To preserve the benefits of the OLS while guarding against spurious findings, an alpha level of 0.01 is used throughout for this analytical technique. The model used is a groups (Up to three levels: Recipients, Decliners, Applicants where applicable) by degree (two levels: master’s or doctoral) by area of study (three levels: SSH, NSE, and HS) analysis of covariance with gender as the covariate.

For some questions respondents are asked to select all that apply in a list of reasons, for example, the question about why they may be ahead of their original plan of studies. For such questions, the Type I error rate of falsely rejecting the null hypothesis was set to 0.05 for the dimension being studied (e.g. within a degree level or within an area of study) with a Bonferroni-type adjustment by requiring the test of each item in the question to reach p < (c2 = 0.05/ (number of items). This is an approximation because the items are not independent from one another.

B.4.3 Challenges and limitations

The 2014 survey included participants who had applied for their scholarship from as far back as 2002, i.e. 12 years earlier. As can be expected, the response rate from distant participants is lower due to loss of contact and reduced interest. Similarly, unsuccessful applicants tended to respond less, possibly due to loss of contact but maybe due to disappointment as well. The weighting scheme partially alleviates the unbiased difficulties associated with loss of contact and re-establishes population-like proportions. However, no amount of weighting can remove bias of opinion.

There are numerous factors influencing people’s answers to survey questions. Most studies take care to include the factors that immediately come to mind as they relate to the primary subject of the research. This is why degree level and area of study figure prominently in this design. However, many extraneous factors are also at play and it is valuable to statistically control for as many of those as possible. In doing so, we are limited by the availability of data. This is why it was possible to include gender as a covariate but unfortunately not age, minority status and presence of dependents. The most unfortunate is that grade point average was only available (in self-reported form with the limitations that this implies) for recipients and not for applicants. Given the crucial importance of GPA as a selection criterion, it would have been very interesting to control for this on questions related to prestige and similar indicators.

There were three groups in the study: recipients, decliners and applicants. The decliners were a very small group, further reduced in size by virtue of having a lot of missing responses. Also not all the questions were presented to this group. Furthermore, it is the comparison between recipients and applicants which is the focus of the study. For these reasons, decliners were not included in this report.

Finally, even without the decliners, the two groups by two degrees by three areas design is complex. In addition to main effects of groups, degrees and areas, it also embodies interactions of group by degree, group by area, and group by degree by area. These higher order interactions were included in the mathematical model to have maximum precision on the main effects. However, the higher order interactions are not very useful in practical terms for the management of scholarship programs and do not necessarily answer interesting questions for the relevance of the program. Therefore, the standard presentation of most results is limited to a focus on:

• Groups
• Degree
• Areas
• Groups within degree
• Groups within area

Appendix C: Outcomes by Degree Level and Study Area

Table C1: Average Number of Paid Hours per Week

Table C2: Average Number of Unpaid Hours per Week

Table C3: Proportion Who Have Not Yet Completed Their Studies and are Continuing in Their Program, CGS Recipients vs. Applicants

Table C4: Distribution of Program Duration (%) by Cohort, Master’s Applicants

Table C5: Distribution of Program Duration (%) by Cohort, Master’s Recipients

Table C6: Distribution of Program Duration (%) by Cohort, Doctoral Applicants

Table C7: Distribution of Program Duration (%) by Cohort, Doctoral Recipients

Table C8: Results from Research, by Degree Level and Area of Study

Table C9: Intended Employment Sector After Degree Program Completion

Table C10: Respondents’ Positions at the University by Cohort, CGS Recipients and Applicants

Table C11: Respondents’ Positions in Other Sectors by Cohort, CGS Recipients and Applicants

Table C12: Relative Advantages of CGS and Tri-Agency Fellowships (SSHRC Doctoral Fellowships)

Date modified:

2017-02-02