Performance Measurement and Evaluation Framework

Executive summary
Introduction: context and objectives
Information sources and methods
Program profile
Risk management
Evaluation issues and questions
Performance monitoring and and evaluation strategy
Footnotes

Executive summary

Context and aims

The Canadian Institutes of Health Research (CIHR) and its partners announced the Strategic Training Initiative in Health Research (STIHR) in the spring of 2001. With a proactive, strategic and problem-driven focus, the STIHR represents an innovation for Canadian health research training support, and a departure from previous applicant- driven models. The STIHR represents an important investment by CIHR and its partners in health research training, with CIHR and the Institutes contributing close to 85$ million over six years. This document outlines a performance measurement strategy and an evaluation framework for the STIHR program. It has two main aims: 1) to provide a structure for performance monitoring of the STIHR, 2) to orient and frame a formative evaluation, to be conducted after three years of implementation, and a summative evaluation, to be completed by the end of the first six-year granting period.

Information sources

This performance measurement and evaluation framework was developed from several sources of information:

Review of relevant documentation;
Qualitative consultations with 22 stakeholders, including: CIHR program personnel; partner agency representatives; grant recipients; grant recipients with partner funding; Program Advisory Committee members; non-funded applicants; CIHR Institutes; Review Committee members; deans of graduate studies; provincial graduate education body representative; and funded trainees.

Program profile

Development of the program profile, including the logic model, led to the identification of program inputs, activities, outputs, short-term outcomes and long-term impacts expected for the STIHR. The latter are::

increased overall (i.e., among CIHR's and partners' research communities) Canadian capacity to successfully address important questions in all areas of health research;
increased capacity of all partners in knowledge application;
and increased attractiveness and prestige of health research careers, including careers outside traditional settings.

Risk management

Areas in need of risk management were identified, i.e., those most likely to prevent or impede the STIHR achieving its objectives, as were strategies to potentially mitigate these risks. Four risk areas are identified as carrying high risk of significant or moderate potential impact, and therefore requiring some or extensive management effort:

Risk of review and selection processes failing to adequately operationalize the intents of the program, specifically of failing to resolve the tension between strategic and capacity development focus and excellence in research training support;
Risks of failure of the partnership model
Risk of inadequate supply of trainees;
Risks that host institutions are not able to adjust and accommodate transdisciplinary, cross-setting training models.

Evaluation issues and questions

The evaluation issues and relevant sub-questions were identified through the consultations and the document review. The issues are:

Implications of STIHR selection processes for the potential success of funded STIHR programs and trainees;
Success of the STIHR in providing integrated national training environments to groups of students forming critical masses in strategic health research areas;
Success of the STIHR in producing its desired outcomes, compared to previous and existing alternative funding mechanisms for health research training;
Success of the partnership model;
Impacts of the STIHR compared to previous and existing alternative funding mechanisms for health research training.

Performance measurement and evaluation strategy

The ongoing performance measurement strategy for the STIHR will use a set of key indicators measuring several of the outputs and outcomes identified in the program's logic model, including the:

implementation and sustainability of the training programs,
the number and quality of participating trainees, and
the quality of the training environments.

Annual performance reports will be produced starting in 2004.

The mid-term and year-six review will address the evaluation issues and questions, using multiple methodologies, including:

Review of financial reports and progress reports submitted in line with the Performance Measurement Strategy;
Surveys of grantees;
Survey of graduate deans (or equivalent);
Survey of Institutes and partners;
Controlled study of trainees: comparing the outcomes of trainees supported through the STIHR against those of trainees supported by traditional funding mechanisms through CIHR or other agencies, most particularly to students funded through national training awards and those funded through research grants;
Thesis and publication review: comparative study of STIHR and non-STIHR-supported students' work.

The mid-term evaluation will be completed by March 2004 and the six-year review will be completed by December 2006.

1. Introduction: context and objectives

The Canadian Institutes of Health Research (CIHR) and its partners announced the Strategic Training Initiative in Health Research (STIHR) in the spring of 2001. The STIHR represents an important investment by CIHR and its partners in health research training, with CIHR and the Institutes contributing close to 85$ million over six years. Aiming to build a national culture of creativity, innovation and transdisciplinary research in the next generation of health researchers, the Initiative funds groups accomplished health researchers to develop and offer health research training programs in their areas of expertise. With a proactive, strategic and problem-driven focus, the STIHR represents an innovation for Canadian health research training support, and a departure from previous applicant- driven models.

This document outlines a performance measurement strategy and an evaluation framework for the STIHR program, in line with the requirements of Results-based Management and Accountability Frameworks (RMAF). RMAFs are intended to provide program managers with tools for: understanding the roles and responsibilities involved in delivering the program; specifying the logical relationships between what it aims to achieve and how it aims to do so; assessing and adjusting its ongoing performance, and ensuring adequate assessment and reporting of its results¹. The development of this performance measurement and evaluation framework thus has two main aims: 1) to provide a structure for performance monitoring of the STIHR, 2) to orient and frame a formative evaluation, to be conducted after three years of implementation, and a summative evaluation, to be completed by the end of the first six-year granting period. The mid-term findings are to be used to adjust and improve program processes based on early experiences, while the aim of the year-six evaluation is to assess the achievements and results (both direct and indirect) of the STIHR in light of its mandate and objectives.

A Performance Measurement and Evaluation Framework Steering Committee oversaw the development of the evaluation framework. Its 10 members represented all program stakeholders, including CIHR, the Institutes, partner organizations, grant recipients, and university administrators. The Committee membership is listed in Appendix 1.

While the assessment of the progress of individual grants remains the purview of the program managers, it is expected that the performance measurement strategy and evaluation framework will identify progress indicators that can be included in funded programs' annual progress reports and rolled up into the performance measurement and evaluation processes at the program level.

2. Information sources and methods

This performance measurement and evaluation framework was developed from several sources of information:

Review of relevant documentation: including the STIHR and Institute program announcements and application guides, CIHR peer review guides, the CIHR evaluation framework, and the evaluation plan for the Aboriginal Capacity and Developmental Research Environments (ACADRE) program.

Consultations with stakeholders: Twenty-two semi-structured interviews were conducted, by telephone or in-person, with program stakeholders within CIHR and the health research community. The sample of interviewees is summarized below.

Table 1: Interviews conducted by type of interviewee

Type	No.
CIHR Program personnel	2
Partner agency representatives	1
Grant recipients	3
Grant recipients with partner funding	1
Program Advisory Committee members	1
Non-funded applicants	2
CIHR Institutes	5
Review Committee members	2
Deans of graduate studies	2
Provincial graduate education body representative	1
Funded trainees	2
Total	22

Interviewees were asked to: indicate their understanding of the program objectives; describe their view of the components of the logic model; identify risk elements and risk management strategies, suggest key performance indicators; and identify and prioritize evaluation issues The interviews were conducted in English or French, according to the respondent's preference. They lasted between 40 and 90 minutes. The interview guide was reviewed by the project manager and pretested, resulting in some minor changes. A second, shorter version was used for key informants not directly involved with the program on an ongoing basis.

3. Program profile

3.1 Overview

The STIHR was designed with the overall aim of creating opportunities for successful research teams to strengthen and expand training in cutting-edge research domains by offering training support for health researchers in targeted areas. Although this model has existed for a number of years in the United States and elsewhere, in Canada, federal health research training was still largely following an applicant-driven model, where students would apply to programs according to their interests, but where programs had little or no leverage to encourage entry into key areas using strategic funding support. The program focuses strongly on transdisciplinarity, defined as "the integration and transformation of fields of knowledge from multiple perspectives in order to define, address and resolve complex real-world problems"². Specifically, the STIHR's objectives are to:

1. Increase the capacity of the Canadian health research community, including areas where it can be demonstrated that there is a need to develop capacity.

2. Enable recruitment and retention of highly qualified individuals from Canada and abroad to undertake health research training in Canada.

3. Support the development of innovative, effective, transdisciplinary, and internationally competitive training programs

4. Engage new mentors and educators in the development and evolution of training strategies.

5. Encourage programs that:

embrace diverse research disciplines and methodological approaches to resolve major health issues and scientific challenges;
integrate training and discussion on the ethical conduct of research and related ethical issues;
develop and measure the individual's communication, teamwork, and leadership skills, i.e., grant writing, peer review; and
incorporate effective research strategies that translate knowledge into practice³.

The STIHR is jointly funded by CIHR, the 13 CIHR Institutes⁴ and key partners. In the first wave of funding, the Institutes identified priority research areas and training program elements, to which applicants could respond. Some of these have made additional funds available through Institute-specific Strategic Training Initiatives⁵, either through expansion of funded programs or by funding additional programs. Applications may also be made outside these areas, addressing other needs in health research training.

The STIHR's key partners have provided funding to initiatives in their jurisdictions or in their content areas. The partners are:

Heart & Stroke Foundation of Canada
Alberta Heritage Foundation for Medical Research
Arthritis Society
Canadian Health Services Research Foundation (CHSRF)
Cancer Care Ontario
Cancer Research Society
Fonds de la recherche en santé du Québec
Institut de recherche Robert-Sauvé en santé et en sécurité du travail (IRSST)
Michael Smith Foundation for Health Research
REPAR (Réseau provincial de recherche en adaptation-réadaptation du Québec)/ IDRPQ (Institut de réadaptation en déficience physique du Québec )

Like most other CIHR-based programs, the STIHR has followed a two-stage application process, with letters of intent (LOI) submitted and reviewed for responsiveness to program eligibility criteria, and a subset of applicants invited subsequently to submit full applications. Each of these may access a small development grant ($5,000) to help support the preparation of the full application.

The first STIHR competition, held in July 2001, attracted 235 letters of intent (LOIs). One hundred and thirty-one teams were invited to submit a full application, of which 127 did so. Fifty-one grants were awarded in March 2002, involving team members from 75 different universities, teaching hospitals and research institutions from across the country. Table 2, below, shows the contributions of the different funding sources to these 51 grants: CIHR funded 14 of these (27%) (the 14 receiving the highest ratings in peer review⁶) while the Institutes and the partners funded the remaining 37 (73%). One of the implications of this decision, to be discussed more fully in the risk assessment chapter, is that there now seem to be two divergent views of the STIHR program: 1) that all funded programs met the criteria for excellence, with some support being provided directly from CIHR and some through the Institutes, with no distinction between the two funding flows; 2) that the programs funded through the Institute were intended to develop capacity in specific research areas, such that excellence did not necessarily have primacy over capacity development needs as seen by Institutes.

Table 2: Funds and grants awarded, STIHR, 2001 competition

Funding source	Funds contributed (2001 competition)	No. of grants funded (2001 competition)
CIHR	$4 m	14
CIHR Institutes and partners	$10 m	37
Total	$14 m	51

The amounts contributed by individual partners are not available in CIHR documents but can be found in some, but not all, partners' financial statements⁷.

A second CIHR-wide competition process was begun in late 2002, with 71 letters of intent received, and 66 invited to submit full applications. These are currently under review.

The merit review team for these applications involved a total of 48 individuals, of whom 30 were members of the letter of intent review committee as well as one of the two merit review committees. Ten review team members were from research settings outside Canada. Nominations to the review team followed the usual CIHR process. Due to time constraints, input on nominations was not systematically sought from partners.

Funded training initiatives are to consist of: "a group of excellent mentors/educators, accomplished in health research, who work collaboratively to offer a research training program of defined format and content to a groups of trainees."⁸ The training programs are encouraged to involve more than one institution and to facilitate transdisciplinary training opportunities. The training programs may include: introduction of undergraduate and professional students to health research; post-graduate research training; and post-doctoral research training, including training or retraining of health professionals, researchers or academics at any career stage.

The recruitment processes for trainees within STIHR programs are still being developed, but will likely involve a combination of open competitions and targeted recruitment.

Seventy percent (70%) of program funds are to be used to support trainee stipends and travel for mentors and trainees between locations; these funds may be supplemented by funds form partners organizations. Other allowable expenses include: costs of developing and coordinating program structure, format and content; development and dissemination of educational materials; advertising and trainee recruitment costs, and other costs deemed justifiable.

The amounts awarded to individual trainees are based on minimum levels established by CIHR. These levels, along with the amounts awarded in CIHR's Research Personnel programs (individual applicant-driven training awards), are shown in Table 3. STIHR teams have the discretionary power to offer and pay trainees a higher stipend than the minimums in Table 3, with no maximum levels being set.

Table 3: Stipend amounts, STIHR and other CIHR training awards

Level	STIHR trainees awards -minimum	CIHR Research Personnel awards
Master's level awards	17, 000	18,000
Doctoral level awards	17,000	20,000
Post-doctoral fellowships	35,000	38,000
Fellowships for health professionals (with no licensure and < 2 years post graduate academic experience)	35,000	38,000
Fellowships for health professionals (With licensure and > 2 years post graduate academic experience)	45,000	47,500

Each program is required to create a Program Advisory Committee or equivalent governance structure, including relevant stakeholders and to which the program is accountable for its progress and future directions.

3.2 Program Logic Model

A logic model for the STIHR program, based on all available sources of information, is shown on page 15. The components of this model are:

Inputs: defined as the program's human, financial and material resources, and their organization. Note that the logic model includes both elements directly managed by CIHR, as well as other critical inputs from external sources. These latter inputs are indicated by dashed boxes;
Activities: the programs, services, products, transactions that transform the inputs into outputs;
Outputs are the direct productions of the program, or the transformation of resources into something that is delivered to clients or users.
Program outcomes: in the short term, the results of the program and their effects on target groups, stakeholders and their environment, and in the medium to long term their impacts on the overall goals of the program.

These components are described in detail below.

Inputs

From partners (external input): In addition to funding (See Table 2), partner organizations provide access to expertise and research networks within defined geographical locations or content domains, as well as access to application settings. They contribute to the strategic orientations of training programs within specific content domains. Most importantly from their perspective, their inputs provide leverage on research training within their locations or domains;
From CIHR: Over and above the grant funds, CIHR coordinates and supports the review processes, and provides infrastructure and policy support to CIHR Institutes. It also provides administrative resources: program managers and officers, technical and clerical support, evaluation, monitoring, internal audit and research ethics functions, as well as links to other agencies and programs.
From CIHR Institutes: As shown in Table 2, the 13 Institutes provide a large portion of the STIHR's funding. Over and above this contribution, the Institutes provide key input to the orientation of the training programs, by identifying areas where researchers are or will be needed, and providing frameworks for the strategic orientations of the training programs in their areas. The Institutes contribute to review panel orientation and support, and may provide administrative resources in the development of the training programs. They also provide links to key partner organizations.
From awardees and their programs (external input): Inputs from the awardees of course include the time, expertise and experience they contribute to training and mentoring. They also include the time and expertise provided by the Program Advisory Committee (or equivalent) in assessing progress and providing future orientations.
From participating research/educational institutions (external input): The participating institutions (universities, teaching hospitals and other types of research centres) provide key inputs in the form of ongoing salary, benefits, physical infrastructure (offices, laboratories) and support to grant recipients. They may also provide some types of administrative and clerical support for the funded programs (e.g. human resource functions, financial services). As part of their ongoing contribution to the broader research community, the institutions also contribute resources to support the merit review process, through salary-supported time for review panel members.

At another level, the participating institutions provide the educational infrastructure through which the training programs operate: this includes the processes and policies devoted to accrediting and integrating courses and curricula, internships and field placements, and other forms of learning. When the training programs involve cross-departmental, cross-faculty and/or cross-institutional collaboration, the institutions must enable comparability and compatibility within their own structures and jurisdictions.

From post-secondary education system (external input): Finally, it is important to recognize that inputs to the STIHR are required from the post-secondary education systems of all Canadian provinces and territories, as well as from centres around the world that may contribute trainees to the programs. The most important of these is an ongoing supply of adequately trained first-degree graduates who are interested in health research as a career. Underlying but nevertheless critical inputs from the post-secondary systems are the infrastructure and mechanisms for development and support of accredited post-secondary education within their jurisdictions – as these determine the requirements for accredited training programs, the STIHR training programs cannot be developed in isolation from them.

Activities

of Strategic Training Initiatives: The central activities of the STIHR program are of course the activities of the funded training programs, including:
1. recruitment, selection and supervision/mentoring of trainees;
2. development and implementation of transdisciplinary/trans-setting training content and mechanisms (curricula, courses, discussion groups etc);
3. development and implementation of transdisciplinary interaction mechanisms among members of the training program and with other relevant groups.
of trainees: Trainees activities are also key : their successful participation in all aspects of the training programs is necessary for the programs to produce their main outputs. Their participation has several dimensions: a strictly academic dimension, a dimension relating to the acquisition of practical, applicable skills, and an interactional dimension. The latter is an important feature of the STIHR program, in that it provides opportunities for trainees to interact with peers and with other researchers in transdisciplinary contexts.
of CIHR, Institutes and partner organizations: CIHR, the CIHR Institutes and the participating partner organizations support the training programs through program management activities: selection processes, financial disbursement, notification and communication, progress and program monitoring and evaluation. The institutes and partners also support trainees by integrating the training programs into their ongoing activities and orientations.
of institutions: The participating institutions are active in the programs through their support to educational and academic program development and support, and through personnel, financial and physical plant management services provided to the training programs. Even more importantly, because the successful implementation of transdisciplinary training programs will be to some extent dependent on cultural shifts within graduate and post-graduate programs to accommodate trans-disciplinary training elements, the participating institutions may be engaged in activities to support these cultural shifts – for example, the creation of "Interdisciplinary Studies" graduate degrees, or cross-departmental committees.

Outputs. The expected outputs for the STIHR program have been organized into three types: partnership outputs, training outputs and training infrastructure outputs.

Partnership outputs: These outputs are the "products" of partners' involvement in the STIHR, over and above the training and infrastructure outputs described below. Partners expect that their role in the STIHR will produce:
1. Increased opportunities for health research training within their domains (i.e. content areas or provinces), as well as leverage to influence the nature and orientations of training in those domains;
2. Sustainable partnerships that are seen as mutually valued and mutually beneficial by CIHR, Institutes and partners
3. In the case of partnerships with content-driven organizations (e.g. Heart and Stroke Foundation), links between trainees and knowledge transfer communities
Training outputs: These outputs are the "products" of the training programs in terms of the trainees' experiences. Expected training outputs can be considered according to all phases of trainees' participation in the programs:
1. Adequate numbers of the highest-possible quality students are recruited, with appropriate gender, language and regional balance;
2. Appropriate supervision and mentoring are provided to trainees, as part of an integrated, ongoing program, such that trainees are fully integrated into the overall transdisciplinary program
3. The trainees proceed with due timeliness through the training programs, resulting in graduates with post-graduate degrees and post-doctoral qualifications in strategic health research domains. (It is important to note that most programs will not begin producing graduates in significant number until at least the third year of funding.)
4. Students are provided with and act on opportunities to interact with multiple relevant research and application networks, across the transdisciplinary spectrum.
Training infrastructure outputs: These refer to changes at the level of the training environments, which the STIHR intends to transform in some fairly fundamental and permanent ways. This includes the implementation and sustaining of transdisciplinary, cross-cutting (cross-departmental, cross-faculty, cross-institutional) and/or partnered (i.e., with partner organizations) training content (e.g. course, curricula, degree programs) and mechanisms (e.g. web-based courses, on-line learning groups). These expected outputs also include permanent transdisciplinary interaction mechanisms: ways for trainees to access formal and informal opportunities to interact with other trainees and research communities (such as conferences, colloquia, summer schools, etc).

Differences between STIHR and other training award program outputs. Although on one level, the outputs described above are similar to those produced by any training award program, some important qualitative and quantitative differences are expected. These are summarized in the table below.

Table 4: Summary - Expected differences in STIHR outputs

**Partnership outputs**
Output	Expected difference between STIHR and other training award program outputs
Increased opportunities for health research training within their domains (i.e. content areas or provinces), as well as leverage to influence the nature and orientations of training in those domains	Through the increased resources and leverage accessible through the partnerships, partners are more able to engage in health research training support that meets their own strategic interests
Sustainable partnerships that are seen as mutually valued and mutually beneficial by CIHR, Institutes and partners	The parties involved develop capacity to work together in ways that are of long term mutual benefit
Links between trainees and knowledge transfer communities	Trainees develop closer links with settings that can apply their eventual research results

**Training outputs**
Output	Expected difference between STIHR and other training award program outputs
Adequate numbers of the highest-possible quality students are recruited, with appropriate gender, language and regional balance;	There are increased numbers of high calibre trainees in health research overall
Appropriate supervision and mentoring are provided to trainees, as part of an integrated, transdisciplinary program	Trainees are more likely to be part of integrated, cross?cutting transdisciplinary programs, rather than a single-laboratory, unidisciplinary programs
The trainees proceed with due timeliness through the training programs, resulting in graduated with post-graduate degrees and post-doctoral qualifications in strategic health research domains	There are increased numbers of graduates in strategic research areas Graduates have more opportunities to acquire transdisciplinary knowledge and competence, and have support to use these in their own research
Trainees produce high quality, transdisciplinary research outputs	Trainees produce theses, papers and conference presentations that are of high quality and reflect their transdisciplinary training experiences
Trainees are provided with and act on opportunities to interact with multiple relevant research and application networks, across the transdisciplinary spectrum	Trainees have more knowledge of and opportunities to interact with key teams and individuals in their research areas

**Training infrastructure outputs**
Output	Expected difference between STIHR and other training award program outputs
Transdisciplinary/trans-setting/partnered training content and mechanisms are implemented and sustained	Personnel and systems have been successfully negotiated among multiple departments, institutions and partners to provide new courses, curricula, teaching mechanisms, etc
Permanent transdisciplinary interaction mechanisms: ways for trainees to access formal and informal opportunities to interact with other trainees and research communities	Trainees have supported, ongoing access to new ways to interact with other trainees and research communities

Program outcomes: Several types of expected outcomes of the STIHR have been identified, but it is noted that the program may result in unexpected outcomes that also should be captured in the evaluation process.

Short term research training outcomes: In the short term, i.e., within the first six-year funding cycle, the STIHR expects to result in:
1. Increased number of trainees undergoing health research training in strategic areas (i.e., areas considered to be of strategic importance by CIHR, Institutes and partner organizations)
2. Increased diversity of the disciplinary base of trainees in health research training in strategic areas, drawing in trainees from outside the health sciences
3. Increased quality of trainees in health research training in strategic areas
4. Increased and earlier competitiveness of trainees in strategic health research funding
5. Enhanced (early) transdisciplinary research productivity
6. Increased (more and stronger) linkages of trainees among themselves, with mentors, and with relevant research environments across the country and the world
7. Increased placement of graduates in research and research-using settings including non-traditional environments.
Short-term infrastructure outcomes. Within the first six-year funding cycle, the STIHR expects to result in:
1. increased capacity for researchers and their institutions to leverage other research and training funds to expand and sustain the programs;
2. sustained capacity to provide integrated national training environments to groups of students forming critical masses in strategic health research areas, and
3. sustained capacity to attract and adequately support strategic, transdisciplinary health research.
Medium–term outcomes: Outcomes over the five to 10 years after the awarding of the grants, the outcomes of the STIHR program would be seen in the career trajectories of trainees, in the quality and impact of the trainees' research results, and in the policy and practice environments relevant to the research results. Specific medium term outcomes would include:
1. Improved health research career trajectories of graduates: increased proportions of graduates going on to develop successful careers in the health research domains they were trained in, resulting from both the quality of their research and their connections to key researchers and settings in their fields. It is important to note however, that these results are predicated on accompanying cultural shifts within research institutions, valuing and rewarding transdisciplinarity in promotion and tenure criteria.
2. Improved career opportunities and outlook for health researchers: more broadly, the STIHR is expected to result in an employment environment more supportive of health research, and providing greater opportunities to engage in and advance through personally and professionally rewarding careers in health research;
3. Increased Canadian capacity to anticipate and respond strategically and innovatively to health research needs: as new health issues and research needs emerge, Canada's research cadre will be better equipped to respond quickly, innovatively and effectively, both in terms of the transdisciplinary knowledge and skills that will be required but also in terms of their stronger national and international networks;
4. Enhanced long-term transdisciplinary, leading edge research productivity and competitiveness: overall, the STIHR should contribute improved Canadian competitiveness (in terms of attracting research support, developing high-quality teams) and productivity (in terms of research outputs and knowledge translation into policy and practice.)
Long-term outcomes: In the long term, up to 20 years after the inception of the STIHR program, its impact should be manifested in contributions to the achievement of the overall goals:
1. Increased overall (i.e., among CIHR's and partners' research communities) Canadian capacity to successfully address important questions in all areas of health research;
2. Increased capacity of all partners in knowledge application;
3. Increased attractiveness and prestige of health research careers, including careers outside traditional settings.

The logic model presented on the next page illustrates the relationships among the inputs, activities, outputs and outcomes of the STIHR, in relation to its overall goals.

Figure 1: Strategic Training Initiative in Health Research - Program Logic Model [ PDF (36 KB) | Help ]

4. Risk management

This framework also incorporates a risk assessment component from a risk-based audit perspective on program review⁹. This component of the framework aims to identify program elements that are most at risk, i.e., those most likely to prevent or impede the STIHR from achieving its objectives.

4.1 Risk criteria

The risks identified below are derived from stakeholders' assessments. During the consultations, stakeholders were asked to identify and discuss the elements of the program they felt to be at greatest risk of preventing or impeding the STIHR from achieving its objectives. This permits linking the risk areas to the causal sequence identified in the logic model, helping therefore to locate potential strategies for mitigating the risks at the appropriate level of intervention (inputs, activities, outputs etc., and/or the links between them).

4.2 Areas of risk and mitigating strategies

Stakeholders identified several areas where STIHR may be at risk for not achieving its objectives. The Table below shows the areas of risk most often identified, their level of risk, their location in the logic model (indicated in Figure 1 by the symbol R), as well as the strategies suggested by stakeholders to mitigate them. For some of these, there was strong consensus among stakeholders, despite the wide range of internal and external perspectives they represented. For others, however? notably Risk 2, stakeholders' views were more mixed, seeming in fact to capture the tension that exists in the health research funding community about this issue.

Note that a number of the risks identified by stakeholders can be seen as related to an overall understanding of the STIHR as an integral part of a large research ecosystem, where the interdependence of components means that significant change in orientation can only be accomplished if all other relevant components are aligned and supportive (and conversely, if none are working at cross-purposes). For example, Risk 6, related to the availability of research funds, underscores the importance of additional research funds to support the increased number of trainees and the increased number of researchers than they will eventually become. Risk 5b can be seen as a concern that significant changes in the orientation of health research training cannot occur unless transdisciplinary merit review committees, as the bellwethers of change, have successfully appropriated the new model.

Table 5: Risk Assessment

Program element at risk	Logic model location	Potential mitigating strategies	Level of risk
1. Risk that the STIHR is under-resourced to make a significant difference in research training capacity Given the long-term consequences of persistent underfunding of research in the last two decades, some stakeholders expressed concern that while the STIHR is a step in the right direction, it does not have the resource levels either within individual programs (to attract the best research trainees to Canadian programs while maintaining a critical mass) or overall (total number of awards, duration) to redress the existing deficit, nor does it carry assurances of the required long-term sustainability.	Inputs	Stakeholders mentioned two obvious mitigating strategies: 1) increased resources in the health research training portfolio (CIHR and its partners, across Canada,) and 2) early development of strategies to ensure sustainability and discourage complacency about continued grant-based support. Note that this was a not key concern for all stakeholders, and some wanted to see evidence of effectiveness before reviewing funding levels.	Low
2. Risk of failing to resolve the tension between the two divergent views of the STHIR program: strategic, capacity-building focus versus excellence in research training support. This refers to the divergent view mentioned earlier, where the STIHR is session as 1) funding only the best programs/trainees, or b) as a capacity development tool for fostering the growth of excellence in areas where capacity is lacking, which may justify the funding of programs and trainees who will not necessarily (yet) meet standard of excellence¹⁰.	Inputs	First, it was clear from the consultations that this an a somewhat emotionally charged issue, and one therefore not likely to be easily dealt with when personal and professional stakes are high – as when funding decisions are being made. Second, mitigating strategies suggested by stakeholders depended on their position within this debate, but fell into two camps: 1) re-affirm the principle of excellence above all others or 2) encourage transparence and dialogue to develop a fully reconciled model of excellence within capacity-development-aimed, strategic focus.	High
3). Risk of failing to develop mechanisms to reconcile different disciplinary world views as they impact on peer review processes in multidisciplinary settings. Stakeholders familiar with the merit review processes identified this risk in different ways, but a common thread was the difficulty in establishing a common ground to assess programs with very different aims and approaches. One result of this difficulty was an apparent difference in the standards used to rate programs in the four pillars, and concern among some researchers that their pillars were losing relative ground in the overall funding picture¹¹. According to some, these inter-disciplinary differences are far more profound and subtle than differences in calibration of rating scales- they arise from differing core values and worldviews among the CIHR pillars.	Inputs	Some stakeholders suggested that in the case of programs with new orientations, such as the STIHR, if committee members could meet prior to the review sessions and develop a common understanding of the review criteria and, more broadly, of the meanings of excellence in transdisciplinary research contexts. Others pointed out that the development of effective transdisciplinarity in merit review processes might require specific supports to merit review committee chairs and members. CIHR and its partners may have already begun to address these issues, through the mandate and activities of the Working Group on Programs and Peer Review¹²	Low: being addressed
4. Risk of inadequate supply of trainees This is an issue that arises out of stakeholders' concerns about the extent to which basic science education systems in Canada are capable of producing adequate supplies of motivated, competent graduates, interested in undertaking post-graduate health research training. Although a dearth was not yet apparent for the programs funded in the 2001 competition, several stakeholders felt that it would be of concern in the next few years. Concern was also expressed about a risk that lack of applicants would lead Institutes to be competing against each other for trainees.	Inputs	This is a systemic problem requiring action at many levels. Some stakeholders argued that CIHR and the Institutes could become more proactive in promoting health research as a career choice, ensuring that undergraduates are aware of and excited by health research as a career possibility. To mitigate against the risk of inter-Institute competition, strategies such as increased flexibility in institutional accountability practices, joint competitions, and pooled recruitment resources should be encouraged.	High
5a). Risks associated with failing to shift from existing applicant-driven models: Risk that programs perpetuate existing models of health research training Several stakeholders consulted, both internal and external, remarked that parts of the research community may be slow to embrace the new paradigm underlying the STIHR. Stakeholders noted four particular types of risk: 1) of continuing to rely on applicant-initiated or "circle-of-friends" approaches to trainee recruitment, failing to be proactive, innovative and open to broader disciplinary sources; 2) of failing to create an innovative transdisciplinary environment for training that provides experiences and opportunities not present in existing programs, i.e., doing the "same old things" (as meritorious as those may be) with the new funds; 3) of using program funds to support trainees who would have been funded through other sources (research stipends in operating grants, personnel awards) -- i.e., merely substituting one source of funding with another, and 4) of mentors failing to invest adequately in mentoring, becoming distracted by competing demands on their time.	Activities-outputs	Overall, the likelihood of these events occurring was felt to be low. Although individual programs may be more or less at risk for failing to make the shift, this should be detectable in annual progress reports – which should result in communication to grantees about it and corrections to their orientations. However, stakeholders who raised this issue agreed that the key mitigating strategy is to ensure that these risks are prevented at the review stage: by ensuring the review criteria clearly operationalize the desired strategic, transdisciplinary, trainee-focused model, that potential applicants are apprised of it and take it seriously, and that performance monitoring tools provide data relevant to this issue.	Low
5b). Risks associated with failing to shift from existing applicant-driven models -- Risk that host institutions are not able to adjust and accommodate transdisciplinary, cross-setting training models. Some stakeholders pointed out that in many ways, host institutions are having to play catch-up to the introduction of the STIHR training model; and while they are supportive of its general orientations, find that it has many profound implications, which will require some time to analyze and deal with. There is therefore a risk that institutions will be slower than needed to accommodate the programs, and resistant to the cultural shift they imply. There is also a risk that trainees will bear the brunt of this lag problem, by being caught between contradictory departmental policies, of having to develop and gain approvals for customized programs, etc.	Activities- outputs	Mitigating strategies suggested by stakeholders included greater involvement of post-secondary education bodies in designing the programs and the merit review criteria, as well as continuing to support high-level discussions between CIHR/ the Institutes/partner organizations and relevant university faculties and forums about the new vision of health research and the role of the STIHR within this. It was also noted that care must be taken to ensure that the mitigating strategies are effective from the trainees' point of view, and not only the institutions': for example, solutions such as the creation of catch-all interdisciplinary PHD degrees may merely displace the problem onto students, absolving the university of the need to reframe its approach to health research training.	High
6. Risk that levels of research support will not be adequate to sustain the greater numbers of health trainees and researchers Although the STIHR intends to increase overall Canadian capacity for health research in key areas, some stakeholders feel there is a risk that the research resources available to the newly-capacitated researchers will not be sufficient.	Medium and long-term outcomes	Stakeholders noted that concurrent adjustments should be made in CIHR's and Institutes', as well as partners' ongoing research support programs, including both operating grants and special initiatives, to ensure that funding levels will met the needs of the new generations of STIHR-trained researchers. Other stakeholders noted however that the STIHR should increase international competitiveness of Canadian researchers, thereby improving access to research funds.	Moderate
7. Risks of failure of the partnership model: There were clear indications from stakeholders that the partnership model initially envisioned may be at risk, with a danger that some partners may withdraw over the short or medium term. As the logic model shows, partners expected that their participation in the STIHR would enhance their own objectives especially in terms of leveraging influence on the orientation of health research training in their communities. However, partners were disappointed by a number of CIHRs' actions – most notably, to fund the top-rated applications directly, which seemed to convey to partners that their interests were secondary.	Activities	It was suggested that CIHR be much more proactive in involving and informing partners about their plans early in the development of programs (i.e., several years before launch), so that 1) partners are able to integrate these in their own planning, consulting with stakeholders etc ) 2) process details which may have major implication for partners may be transparently negotiated with adequate time for this allowed As this risk factor might have been partly a result of having to cope with unwieldy financial management demands coming from the federal government, it was also suggested that strategies developed to minimize the impacts of these on partners.	High

In Figure 2, below, the risk areas have been classified below according to their severity of impact (i.e., how serious the impact of the occurrence would be on the STIHR's capacity to reach its objectives and fulfill its mandate: significant impact, moderate impact, or minor impact) and their likelihood of occurrence (low, medium or high). Risk management actions are shown in Figure 3: Four risk areas are identified as carrying high risk of significant or moderate potential impact, and therefore requiring some or extensive management effort:

2. Risk of failing to resolve the tension between strategic, capacity-building focus and excellence in research training support;
7. Risks of failure of the partnership model
4. Risk of inadequate supply of trainees;
5. Risks associated with failing to shift from existing applicant-driven models: b) Risk that host institutions are not able to adjust and accommodate transdisciplinary, cross-setting training models.

Figure 2 : Risks for the Strategic Training Initiative in Health Research

Impact	RISK MANAGEMENT ASSESSMENT
Significant	1. Risk that the STIHR is under-resourced to make a significant difference in research capacity		2) Risk of failing to resolve the tension between strategic, capacity-development focus and excellence in research training support. 6. Risks of failure of the partnership model
Moderate	3) Risk of failing to develop mechanisms to reconcile different disciplinary world views as they impact on peer review processes in multidisciplinary settings. 5a) Risk that programs perpetuate existing models of health research training	6. Risk that levels of research support will not be adequate to sustain the greater numbers of health trainees and researchers	4). Risk of inadequate supply of trainees 5b) Risk that host institutions are not able to adjust and accommodate transdisciplinary, cross-setting training models.
Minor
	Low	Medium	High
	LIKELIHOOD

Figure 3: Risk management actions

Impact	RISK MANAGEMENT ACTIONS
Significant	Considerable management required	Must manage and monitor risks	Extensive management essential
Moderate	Risks may be worth accepting with monitoring	Management effort worthwhile	Management effort required
Minor	Accept risks	Accept but monitor risks	Manage and monitor risks
	Low	Medium	High
	LIKELIHOOD

5. Evaluation issues and questions

RMAF's should include a set of issues and questions which, "when answered, will allow for a thorough assessment of the relevance, success and cost-effectiveness of a program¹³. A preliminary set of evaluation issues and questions were identified from the interviews and documentation review, many following directly from the risk areas identified in the previous section. They are described below and summarized in Table 6.

Issue 1: Implications of STIHR selection processes for the potential success of funded STIHR programs

This issue focuses on the impacts of the program on the adequacy of its inputs, by directly addressing the question of how the tension between excellence and strategic, capacity-development funding might have affected both the likelihood of success of funded programs, and the quality of the trainees who found themselves part of the program (in terms of productive, innovative research career likelihood):

1.1 What is the nature of the interplay between strategic, capacity-development and excellence-driven funding within the strategic research areas with STIHR funding, and what are the implication of this for overall likelihood of program success?

1.2 What impacts did the selection processes (initiatives and trainees) have on the quality of selected training environments and trainees?

Issue 2: Success of the STIHR in providing integrated national training environments to groups of students forming critical masses in strategic health research areas

This issue is essentially one of implementation, focusing on how well the STIHR is succeeding in providing the training environments and experiences that would be expected to produce the program benefits. (In the logic model, it centres on the "Activities – Strategic Training Initiatives" box). It would address the risk identified in the previous section, that programs are not successfully making the shift from existing, applicant-driven training models assessing the extent to which they are perpetuating existing models of health research training:

2.1 How adequate are the experiences of trainees in terms of key research training components:
Quality, breadth, depth and coherence of multidisciplinary education in all relevant areas of the strategic research focus?
Quality of supervision and mentoring, in terms of supportiveness, utility, and transdisciplinarity ?
Quality and diversity of research skills development experiences?
Quality and frequency of transdisciplinary interactions across multiple research settings, nationally and internationally?

It also examines the risk that host institutions are not able to adjust and accommodate transdisciplinary, cross-setting training models, by examining the adequacy of their actions in support of STIHR program implementation (the "Activities: Institutions" box in the logic model):

2.2 How adequate are the mechanisms and levels of support provided for the development and implementation of integrated, transdisciplinary, national training environments within each program?

2.3 How successfully have programs and their partners been able to negotiate the implementation of their visions of transdisciplinary health research training in the contexts of their institutions and the larger context of graduate research training?

Issue 3: Success of the STIHR in producing its desired outcomes, compared to previous and existing alternative funding mechanisms for health research training

This issue look directly at the success of the STIHR in terms of producing outcomes that should be visible within the first few years of funding, in comparison to existing forms of research training support (personnel awards, research stipends including in operating grants, offered by CIHR and other research agencies). Specifically, it addresses the success of the STIHR in producing the research training infrastructure outcomes identified in the logic model, asking the question:

How successful has the STIHR been in producing:

3.1 Increased number of students in health research training in strategic areas (i.e., areas considered to be of strategic importance by CIHR, Institutes and partner organizations)?
3.2 Increased diversity of disciplinary base of students in health research training in strategic areas?
3.3 Increased quality of students in health research training in strategic areas?
3.4 Increased early competitiveness of students in strategic health research funding?
3.5 Enhanced early transdisciplinary research productivity ?
3.6 Increased and strengthened linkages of students among themselves, with mentors, and with relevant research environments in and outside Canada?
3.7 Increased placement of graduates in research and research-using settings including non-traditional environments?
3.8 Increased capacity to leverage other research/ training funds to support the program?
3.9 Sustained capacity to provide integrated national training environments to groups of students forming critical masses in strategic health research areas?
3.10 Sustained capacity to attract and adequately support strategic, transdisciplinary health research?

Issue 4: Success of the partnership model

The STIHR partnership model was intended to forge new working relationships among CIHR, the Institutes and provincial and NGO partners as important actors in setting and managing the national health research training agenda. This issue will address how successfully this model has been operationalized, from the perspectives of all partners and their respective accountability structures, through the questions:

4.1 How successfully has the STIHR partnership model been implemented and operationalized?

4.2 How has the STIHR partnership model affected all partners' capacities to address their strategic interests through the STIHR?

Issue 5: Impacts of the STIHR compared to previous and existing alternative funding mechanisms for health research training

This issue addresses the STIHR's key medium-term impacts, through the questions:

What has been the impact of the STIHR on:

5.1 career trajectories of funded trainees?
5.2 career opportunities for health researchers?
5.3 Canadian capacity to anticipate and respond strategically to health research needs?
5.4 transdisciplinarity of Canadian health research?
5.5 transdisciplinary research productivity and competitiveness?

However, stakeholders noted that the full impacts of the STIHR program would not be expected to be visible until a) the programs have produced several cohorts of health researchers, and b) the first cohorts are in mid-career – that is, not until 15 to 20 years from the program's inception.

Table 6: Strategic Training Initiative in Health Research: Proposed Evaluation Issues and Questions

	Timing
Issue 1: Implications of STIHR selection processes for the potential success of funded STIHR programs
1.1	What is the nature of the interplay between strategic, capacity-development and excellence-driven funding within the strategic research areas with STIHR funding, and what are the implication of this for overall likelihood of program success?	Mid-term and summative
1.2	What impacts did the selection processes for initiatives have on the quality of selected training environments?	Mid-term
1.3	What impacts did the selection processes for trainees have on the potential success of the STIHR programs?	Mid-term
Issue 2: Success of the STIHR in providing integrated national training environments to groups of students forming critical masses in strategic health research areas
2.1	How adequate are the experiences of trainees in terms key research training components: Quality, breadth, depth and coherence of multidisciplinary education in all relevant areas of the strategic research focus? Quality of supervision and mentoring, in terms of supportiveness, utility, and transdisciplinarity? Quality and diversity of research skills development experiences? Quality and frequency of transdisciplinary interactions across multiple research settings, nationally and internationally?	Mid-term and summative
2.2	How adequate are the mechanisms and levels of support provided for the development and implementation of integrated, transdisciplinary, national training environments within each program?	Mid-term
2.3	How successfully have the programs and their partners been able to negotiate the implementation of their visions of transdisciplinary health research training in the contexts of their institutions and the larger context of graduate research training?	Mid-term
Issue 3: Success of the STIHR in producing its desired outcomes, compared to previous and existing alternative funding mechanisms for health research training
3.1	How successfully has STIHR increased number of students in health research training in strategic areas?	Mid-term and summative
3.2	How successfully has STIHR increased diversity of disciplinary base of students in health research training in strategic areas?	Mid-term and summative
3.3	How successfully has STIHR increased quality of students in health research training in strategic areas?	Mid-term and summative
3.4	How successfully has STIHR increased early competitiveness of students in strategic health research funding?	Summative
3.5	How successfully has STIHR enhanced early transdisciplinary research productivity?	Summative
3.6	How successfully has STIHR increased and strengthened linkages of students among themselves, with mentors, and with relevant research environments in and outside Canada?	Mid-term and summative
3.7	How successfully has STIHR increased placement of graduates in research and research-using settings including non-traditional environments?	Summative
3.8	How successfully has STIHR increased capacity to leverage other research/ training funds to support the programs?	Summative
3.9	How successfully has STIHR sustained capacity to provide integrated national training environments to groups of students forming critical masses in strategic health research areas?	Summative
3.10	How successfully has STIHR sustained capacity to attract and adequately support strategic, transdisciplinary health research?		Summative
Issue 4: Success of the partnership model
4.1	How successfully has the STIHR partnership model been implemented and operationalized?		Mid-term and summative
4.2	How has the STIHR partnership model affected all partners' capacities to address their strategic interests through the STIHR?		Mid-term and summative
Issue 5: Impacts of the STIHR compared to previous and existing alternative funding mechanisms for health research training
5.1	What is the impact of STIHR on career trajectories of funded trainees?	Summative
5.2	What is the impact of STIHR on career opportunities for health researchers?	Summative
5.3	What is the impact of STIHR on Canadian capacity to anticipate and respond strategically to health research needs?	Summative
5.4	What is the impact of STIHR on transdisciplinarity of Canadian health research?	Summative
5.5	What is the impact of STIHR on transdisciplinary research productivity and competitiveness?	Summative

6. Performance monitoring and evaluation strategy

This chapter outlines the proposed strategies for monitoring ongoing performance of the STIHR, as well as evaluation strategies for the mid-term and summative evaluations.

6.1 Evaluation design issues: comparisons and controlled studies

As stated earlier, the mid-term findings are to be used to adjust and improve program processes based on early experiences, while the aim of the year-six evaluation is to assess the achievements and results (both direct and indirect) of the STIHR in light of its mandate and objectives.

To ensure the credibility of the evaluation within the research community, the evidence generated by both phases must be as strong as possible. The rigour of the evaluation will depend on the strength of the inference that can be drawn from the available comparisons between would have happened in the absence of the program, and what has happened with the program in place. We recommend that the performance measurement and evaluation design strategies aim to maximize strength of inference in two ways:

By establishing baseline measures (pre-STIHR, based on the preceding five years) and conducting repeated, regular measurements of key indicators, to track changes over time, in comparison to the baseline.
1. While not a strong evaluation design, these comparisons will at least allow measures of change, as opposed to mere program description. They will be particularly useful for easily quantifiable indicators such as numbers of types of trainees in targeted research areas. However, the challenges and cost of establishing appropriate baselines should not be underestimated.

By conducting well-designed, controlled studies in a few key outcome/impact areas. We recommend two such studies, both of which would compare the outcomes of trainees supported through the STIHR against those of trainees supported by traditional funding mechanisms through CIHR or other agencies, most particularly to students funded through national training awards and those funded through research grants.

Controlled study of students' experiences and outcomes (Evaluation question 2.1). The extent to which the STIHR is producing a measurable difference in students' health research training experiences is obviously key to the success of the program. A controlled study, comparing appropriately representative and comparable samples of students supported through the STIHR and students supported through other, more traditional funding programs as well as students who would not be eligible for funding under other programs (e.g. CIHR personnel awards, research stipends) as well as students who may not have eligible for funding previously, will allow clear understanding of what types of differential impacts the program has had. This study should examine the impacts of STIHR versus other programs on: 1) students' exposure to transdisciplinary research training; 2) impacts of that exposure on the students' own work; 3) opportunities for and impacts of interaction with peers and researcher/teams in other settings; 3) research productivity and quality; 4) intentions to pursue health research as a career and finally but not least; 6) students' satisfaction with their health research training. The study should also assess the levels of financial and other support actually received by students in the two groups. Samples of students in the two groups would be randomly selected balancing across levels, years of entry into program, region, and CIHR pillar (reflecting pillar distribution of funded programs), and measured ideally at three time points: in mid-training, at the point of graduation, and two to five years post-graduation.

Comparative study of the impacts of transdisciplinarity on trainees' research outputs. (Evaluation question 3.5) A central premise of the STIHR is that proving health research trainees with transdisciplinary knowledge and skills will increase the transdisciplinarity and innovativeness of their own work. This may be manifested directly, in terms of the integration of research literatures, techniques and perspectives from other disciplines, or more indirectly, in terms of openness or receptivity to ideas from outside the trainees' core discipline. In order to assess the extent to which, and how, the STIHR does actually influence the nature of trainees' research activities, a controlled study comparing the primary research outputs -- master's thesis, doctoral dissertations, early publications and communication ---- of STIHR trainees and students supported through other, more traditional funding programs (e.g. CIHR personnel awards, research stipends). The outputs would be compared in terms of: 1) quality; 2) quantity; and 3) degree and nature of transdisciplinarity. While the first two variables may use conventional bibliometric techniques, the third will require developing and validating methodologies to assess transdisciplinarity in research output across a variety of disciplines, itself a challenging task. For example, criteria for this review might be established by an independent panel of national and international researchers with expertise in the specific topic areas and grounding in transdisciplinary health research. The review process itself could also be conducted by a panel of independent experts, in a manner similar to blind peer review of journal submissions. To assess feasibility of the project, pilot studies could be conducted in a limited number of strategically important disciplines, for example those included in the biomedical pillar.

For other program impacts, self-assessed changes (by grantees, institutions, institutes and partners) may provide adequate information, although controlled studies could be conducted for other issues as well.

6.2 Overview of data sources

Six main data sources are proposed, including the two controlled studies described above. Some of these are applicable to both the mid-term and final evaluations, and some more suited to the final evaluation.

Review of progress and financial reports: Several key indicators should be available in the annual progress reports to be submitted to CIHR as well as in annual financial reports. These can be reviewed annually and the relevant indicators compiled, in some cases comparing outputs report to outputs intended, as identified in the funding applications.

Survey of grantees: Brief Internet-based surveys of all STIHR awardees would provide access to several indicators, including measures of the quality and quality of training program implementation and transdisciplinary mentoring and supervision.

In addition, a sample of researchers from the rejected submissions at both the letter of intent and full proposal stages should be included to assess the results of non-funded programs in terms of the main outcomes, as they may have been able to achieve some of their proposed objectives in the absence of STIHR funding¹⁴.

Survey/study of trainees: This is the controlled study described above, which would be most relevant for the mid-term and six-year evaluations. Alternatively, cross-sectional surveys of STIHR- supported trainees could be conducted to assess their experiences, particularly as part of the performance measurement strategy¹⁵.

Survey of graduate deans (or equivalent): Indicators of the strength and sustainability of the programs could be obtained through brief telephone or Internet base surveys of graduate deans (or their equivalents in teaching hospitals and other types of research centres).

Survey of Institutes and partners: Institutes and partners will be able to provide important qualitative and qualitative data on the overall portrait of health research training in their research domains, both including and outside the STIHR's contributions, as well as on the success of the partnership model (Issue 4).

Thesis and publication review: comparative study: This study is described above.

6.3 Ongoing performance measurement strategy

The aim of ongoing performance measurement is to monitor the progress of programs toward their expected results. Key indicators are measured regularly and the results compared to expectations, in order to provide program managers with information that can be used to adjust the program in order to ensure optimal performance and attainment of program objectives or signal the need for deeper exploration. The measurement strategy for the STIHR programs summarized in Table 7 identifies a set of key indicators and their data sources for each of the outputs and short-term outcomes identified in the logic model. Responsibility for data collection and reporting of each of the proposed indicators is shown, based on the assumption that several of the proposed indicators could be available through grantees' annual progress reports to CIHR and/or to their Program Advisory Committees.

The priority level for each indicator considered is indicated in the last column. These are based on input from the Steering Committee, aiming to limit the indicator-collection effort to a realistic level.

Table 7: Performance Measurement Strategy for the STIHR

**Outputs**
Program Element	Performance Indicators	Data Sources	Responsibility for data collection and reporting	Frequency of Measurement	Priority
High quality trainees recruited: with gender, language and regional balance	Mean undergraduate standing of trainees admitted to programs Number of disciplines and undergraduate settings represented among trainees Degree to which gender, regional and linguistic balance matches Canadian population	Progress reports	Grantees	Annual	Low
Appropriate supervision/ mentoring provided	Proportion of mentors' time spent in supervision and mentoring Degree of transdisciplinarity of supervision and mentoring: proportion of trainees with multi-disciplinary and/or multi-setting co-supervision; number. of supervisory/mentoring hours provided by each discipline	Progress reports compared to applications	Grantees	Annual	Low
Trainees educated/mentored and graduated	Proportion of programs fulfilling or exceeding training targets (where these exist) Number of trainees in programs, by level Proportion of students graduating	Progress reports compared to applications Financial reports: salaries – student support	Grantees	Annual Annual	Low
Trainees' interactions with multiple relevant research and application networks	Degree of transdisciplinary exposure: Proportion of trainees' research work and course work falling outside traditional disciplinary boundaries No. of occasions and no. of hours of interaction with researchers and research networks outside home setting	Progress reports Survey/study of trainees	Grantees CIHR	Annual At three and six years	High
Transdisciplinary/trans-setting training content and mechanisms implemented and sustained	State of development and degree of implementation of planned training content Degree of assurance of program sustainability	Progress reports compared to applications Survey of grantees Survey of graduate deans	Grantees CIHR	Annual At three and six years At three and six years	High
Transdisciplinary interaction mechanisms implemented and sustained	State of development and degree of implementation of interaction mechanisms Degree of assurance of interaction mechanism sustainability	Progress reports compared to applications Survey of grantees	Grantees CIHR	Annual At three and six years	Low

**Short-term outcomes**
Program Element	Performance Indicators	Data Sources	Responsibility for data collection and reporting	Frequency of Measurement	Priority
Increased number of students in health research training in strategic areas	Numbers of students in targeted research areas given in RFAs as compared to pre-STIHR 5 years	Survey of Institutes and partners	CIHR	At three and six years	High
Increased diversity of disciplinary base of students in health research training in strategic areas	Total number of undergraduate disciplines represented in targeted research areas given in RFAs as compared to pre-STIHR 5 years	Survey of Institutes and partners	CIHR	At three and six years	Low
Increased quality of students in health research training in strategic areas	Mean undergraduate standing of trainees in targeted research areas given in RFAs as compared to pre-STIHR 5 years	Survey of Institutes and partners	CIHR	At three and six years	High
Increased and earlier competitiveness of graduates in strategic health research funding	Amounts of grant funds obtained by program graduates in research positions Diversity of sources of funds obtained, compared to previous sources	Survey/study of students	CIHR	At six years	Low
Enhanced (early) transdisciplinary research productivity	Quantity and quality of trainees' early reports, publications and communications Degree of transdisciplinarity of publications and communications, in terms of perspectives, approaches, methodologies and authors	Progress reports Thesis and publication review: comparative study	Grantees CIHR	Annual At six years	Low
Increased (more and stronger) linkages of students among themselves, with mentors, and with partners and stakeholders in relevant research environments across the country and the world	Number of national and international networking opportunities provided to students Quality of mentorship compared to trainees' expectations Amount of time spent by trainees with researchers outside of their major supervisors and their main disciplinary areas Number of trainees with multi-departmental and/or multi-institutional supervisory committees Amount of interaction among student in training program Proportion of trainees and graduates who are familiar with work of key research teams in relevant research environments across the country and the world Proportion of trainees and graduates who have met and discussed their work with partners and stakeholders in relevant research environments across the country and the world	Survey/study of students	CIHR	At six years	High
Increased placement of graduates in research and research-using settings including non-traditional environments	Proportion of trainees who intend to pursue health research as a career Proportion of trainees who intend to pursue further health research training Proportion of trainees who are in full-time research positions by two years after completion of highest degree	Survey/study of students Survey/study of students Survey/study of students	CIHR CIHR CIHR	At three and six years At six years At three and six years	High
Increased capacity to leverage other research/ training funds to support program	Amounts of grant funds obtained by program participants in targeted research areas given in RFAs as compared to pre-STIHR 5 years Diversity of sources of grant funds obtained by program participants, in targeted research areas given in RFAs as compared to pre-STIHR 5 years	Survey of Institutes and partners	CIHR	At six years	Low
Sustained capacity to provide integrated national training environments to groups of students forming critical masses in strategic health research areas	Amount of funds and faculty hours committed by institutions to sustain programs Amount of funds to sustain programs obtained from other sources	Survey of graduate deans Survey of Institutes and partners Survey of grantees	CIHR	At six years At six years	High
Sustained capacity to attract and adequately support strategic, transdisciplinary health research	Amounts of grant funds obtained by program participants in targeted research areas given in RFAs as compared to pre-STIHR 5 years	Survey of Institutes and partners	CIHR	At six years	Low

In summary, the high-priority indicators, forming the backbone of the ongoing performance measurement strategy for the STIHR would be:

Degree of transdisciplinary exposure: Proportion of trainees' research work and course work falling outside traditional disciplinary boundaries
Quality of mentorship compared to trainees' expectations
Amount of time spent by trainees with researchers outside of their major supervisors and their main disciplinary areas
Number of trainees with multi-departmental and/or multi-institutional supervisory committees
Amount of interaction among students in training program
State of development and degree of implementation of planned training content
Degree of assurance of program sustainability
Numbers of students in targeted research areas given in RFAs as compared to pre-STIHR 5 years
Mean undergraduate standing of trainees in targeted research areas given in RFAs as compared to pre-STIHR 5 years¹⁶
Number and proportion of trainees and graduates who are familiar with work of key research teams in relevant research environments across the country and the world
Number of national and international networking opportunities provided to students
Proportion of trainees and graduates who have met and discussed their work with partners and stakeholders in relevant research environments across the country and the world
Proportion of trainees who intend to pursue health research as a career
Proportion of trainees who intend to pursue further health research training
Proportion of trainees who are in full-time research positions by two years after completion of highest degree
Amount of resources (faculty hours, space, equipment etc) committed by institutions to sustain programs
Amount of funds to sustain programs obtained from other sources.

The data that grant recipients would be expected to provide in their annual progress reports are listed below. An example progress reporting form is shown in Appendix 2.

About trainees:

Undergraduate standing of trainees admitted to programs
Number of disciplines and undergraduate settings represented among trainees
Gender, regional and linguistic makeup of trainee group
Number of trainees in programs, by level, compared to targets (where these exist)
Number of students graduating, by level, compared to targets (where these exist)
Number of trainees' early reports, publications and communications: dates submitted and journals targeted

About the program:

State of development and degree of implementation of planned training content
Degree of assurance of training program sustainability
State of development, degree of implementation and expected sustainability of interaction mechanisms
Degree of transdisciplinary exposure: Proportion of trainees' research work and course work falling outside traditional disciplinary boundaries

6.4 Evaluation Strategy

The evaluation strategy addresses the evaluation issues and questions, drawing on the same basic set of information sources and comparisons. The table below illustrates the main indicators and data sources for all the evaluation questions. In addition, the evaluation should respond to and assess unexpected outputs and outcomes, especially in terms of potential negative impacts such as competition among Institutes.

Table 8: STIHR Evaluation Strategy Summary

**Issue 1: Implications of STIHR selection processes for the potential success of funded STIHR programs**
Evaluation Issues and Questions	Indicators	Data Sources	Evaluation phase
1.1 What is the nature of the interplay between strategic and excellence-driven funding within the strategic research areas with STIHR funding, and what are the implication of this for overall likelihood of program success?	Degree of successful resolution of tension Effectiveness of reconciliation strategies	Survey of grantees Survey of Institutes and partners	Mid-term and summative
1.2 What impacts did the selection processes (initiatives and trainees) have on the quality of selected training environments and trainees?	Quality of STIHR trainees versus others Perceived quality of training programs	Controlled study of trainees Survey of Institutes and partners	Mid-term

**Issue 2: Success of the STIHR in providing integrated national training environments to groups of students forming critical masses in strategic health research areas**
Evaluation Issues and Questions	Indicators	Data Sources	Evaluation phase
2.1 How adequate are the experiences of trainees in terms of key research training components: Quality, breadth, depth and coherence of multidisciplinary education in all relevant areas of the strategic research focus? Quality of supervision and mentoring, in terms of supportiveness, utility, and transdisciplinarity? Quality and diversity of research skills development experiences? Quality and frequency of transdisciplinary interactions across multiple research settings, nationally and internationally?	Quality, breadth, depth and coherence of multidisciplinary education in all relevant areas of the strategic research focus Quality of supervision and mentoring, in terms of supportiveness, utility, and transdisciplinarity Quality and diversity of research skills development experiences Quality and frequency of transdisciplinary interactions across multiple research settings	Controlled study of trainees Survey of grantees	Mid-term and summative
2.2. How adequate are the mechanisms and levels of support provided for the development and implementation of integrated, transdisciplinary, national training environments within each program?	Degree to which required supports to implement the training programs are available Degree to which implementation objectives have been reached	Review of progress reports Survey of grantees Survey of graduate deans	Mid-term
2.3 How successfully have programs and their partners been able to negotiate the implementation of their visions of transdisciplinary health research training in the contexts of their institutions and the larger context of graduate research training?	Extent of host institution acceptance and integration of program training model Extent to which model has influenced or been considered or used as a model in other areas	Review of progress reports Survey of grantees Survey of graduate deans	Mid-term

**Issue 3: Success of the STIHR in producing its desired outcomes, compared to previous and existing alternative funding mechanisms for health research training**
Evaluation Issues and Questions	Indicators	Data Sources	Evaluation phase
3.1 How successfully has STIHR increased number of students in health research training in strategic areas?	Number of students in health research training in strategic research areas defined in RFAs	Controlled study of trainees Survey of Institutes and partners	Mid-term and summative
3.2 How successfully has STIHR increased diversity of disciplinary base of students in health research training in strategic areas?	Total number of undergraduate disciplines represented in targeted research areas given in RFAs	Controlled study of trainees Survey of Institutes and partners	Mid-term and summative
3.3 How successfully has STIHR increased quality of students in health research training in strategic areas?	Mean undergraduate standing of trainees in targeted research areas given in RFAs	Controlled study of trainees Survey of Institutes and partners Comparative study of research outputs	Mid-term and summative
3.4 How successfully has STIHR increased early competitiveness of students in strategic health research funding?	Amounts of grant funds obtained by program trainees Diversity of sources of funds obtained, compared to previous sources	Controlled study of trainees Survey of grantees	Summative
3.5 How successfully has STIHR enhanced early transdisciplinary research productivity?	Quantity and quality of trainees' early reports, publications and communications Degree of transdisciplinarity of publications and communications, in terms of perspectives, approaches, methodologies and authors	Controlled study of trainees Comparative study of research outputs	Summative
3.6 How successfully has STIHR increased and strengthened linkages of students among themselves, with mentors, and with relevant research environments in and outside Canada?	Proportion of trainees and graduates who are familiar with work of key research teams in their area, outside their home setting Proportion of trainees and graduates who have met and discussed their work with of key research teams in their area, outside their home setting	Controlled study of trainees Survey of grantees	Mid-term and summative
3.7 How successfully has STIHR increased placement of graduates in research and research-using settings including non-traditional environments?	Proportion of trainees who intend to pursue health research as a career Proportion of trainees who intend to pursue further health research training Proportion of trainees who are in full-time research positions by five years after completion of highest degree	Controlled study of trainees	Summative
3.8 How successfully has STIHR increased capacity to leverage other research/ training funds to support the programs?	Amounts of grant funds obtained by program participants in targeted research areas given in RFAs Diversity of sources of grant funds obtained by program participants, in targeted research areas given in RFAs	Survey of grantees Survey of Institutes and partners	Summative
3.9 How successfully has STIHR sustained capacity to provide integrated national training environments to groups of students forming critical masses in strategic health research areas?	Amount of funds and faculty hours committed by institutions to sustain programs Amount of funds to sustain programs obtained from other sources	Survey of grantees Survey of Institutes and partners	Summative
3.10 How successfully has STIHR sustained capacity to attract and adequately support strategic, transdisciplinary health research?	Amounts of grant funds obtained by program participants in targeted research areas given in RFAs	Survey of grantees Survey of Institutes and partners	Summative

**Issue 4: Success of the partnership model**
Evaluation Issues and Questions	Data Sources	Evaluation phase
4.1How successfully has the STIHR partnership model been implemented and operationalized?	Survey of Institutes and partners	Mid-term and summative
4.2 How has the STIHR partnership model affected all partners' capacities to address their strategic interests through the STIHR?	Survey of Institutes and partners	Mid-term and summative

**Issue 5: Impacts of the STIHR compared to previous and existing alternative funding mechanisms for health research training**
Evaluation Issues and Questions	Indicators	Data Sources	Evaluation phase
5.1 What is the impact of STIHR on career trajectories of funded trainees?	Proportion of STIHR trainees in full-time, stable research positions within five years of graduation	Controlled study of trainees	Summative
5.2 What is the impact of STIHR on career opportunities for health researchers?	Number of new health research positions opened in targeted research areas given in RFAs	Survey of Institutes and partners	Summative
5.3 What is the impact of STIHR on Canadian capacity to anticipate and respond strategically to health research needs?	Perceived Canadian capacity to anticipate and respond strategically to health research needs	Survey of Institutes and partners	Summative
5.4 What is the impact of STIHR on transdisciplinarity of Canadian health research?	Perceived transdisciplinarity of Canadian health research	Survey of Institutes and partners Comparative study of research outputs	Summative
5.5. What is the impact of STIHR on transdisciplinary research productivity and competitiveness?	Amounts of grant funds obtained by Canadian researchers in targeted research areas given in RFAs Research productivity of STIHR participants	Survey of Institutes and partners Controlled study of trainees	Summative

6.5 Reporting plan

The following plan is to ensure that the results of ongoing performance measurement and evaluation are reported and that reporting commitments are met.

Overall, the reporting strategy for the framework should consist of statutory, annual reports to the following:
- CIHR Executive Committee and Governing Council
- CIHR Institutes: their management and Advisory Boards
- partner agencies: the management and governance structures
- each participating program and its Program Advisory Committees
- institutions hosting each of the programs.

Reports on the mid-term review and year-six should also be made to each of the above.

Table 9 summarizes the Reporting Plan for the Performance Measurement and
Evaluation Strategy.

Table 9: Reporting Plan Summary

Activity	Report	Reporting dates	STRONG>Responsibility for reporting
Ongoing performance measurement	Annual STIHR performance report	March 2004 March 2005 March 2006 March 2007 March 2008	CIHR- STIHR program
Mid-term review	Interim evaluation report	March 2004	CIHR Evaluation Unit
Year?six review	Final evaluation report	September 2006	CIHR Evaluation Unit

Footnotes

Treasury Board Secretariat, Guide for the Development of Results-based Management and ACcountability Frameworks, August 2002.
http://www.cihr-irsc.gc.ca/services/funding/opportunities/institutes/2001/trans_f.shtml
CIHR Strategic Training Program Grants 2002 Competition, Instructions for the Full Application, p. 1
Institute of Aging (IA), Institute of Aboriginal People's Health (IAPH), Institute of Cancer Research (ICR), Institute of Circulatory and Respiratory Health (ICRH), Institute of Genetics (IG), Institute of Gender and Health (IGH), Institute of Human Development, Child & Youth Health (IDCYH), Institute of Health Services and Policy Research (ISPR), Institute of Infection and Immunity (III), Institute of Musculoskeletal Health and Arthritis (IMHA), Institute of Nutrition, Metabolism and Diabetes (INMD), Institute of Neurosciences, Mental Health and Addiction (INMHA), Institute of Population and Public Health (IPPH).
Executive Committee meeting minutes, March 1 2002. http://www.cihr-irsc.gc.ca/publications/about_cihr/meeting_minutes/ governing_council/committees/executive/ exec_minutes_7_e.shtml
For example: http://www.msfhr.org/docs/MSFHR_Audited_Financial_Statements.pdf; showing a contribution of $54,000 in 2001/2002.
Request for Applications. CIHR Strategic Training Initiative in Health Research, March (?) 2001.
Treasury Board, April 2001, Integrated Risk Management Framework
CIHR's Governing Council is well-aware of this issue and has considered how to address it: It was suggested that Governing Council must clarify its intentions with respect to a common basis for funding, and subsequently, how this decision will impact capacity building and the promotion of excellence. (CIHR Executive Committee minutes, March 1, 2002); The research community may be receiving mixed signals about stringency of adjudication process for open competitions and strategic initiatives as a result of funding the latter at a lower cut-off than that of open competitions, thus potentially suggesting that less that excellent research is being funded by CIHR (CIHR Governing Council Minutes, March 11 2002).
CIHR stakeholders are already well-aware of this issue, and internal analyses were undertaken comparing mean ratings of funded and all applicants across pillars.
Meeting Minutes, Working Group on Programs and Peer Review, November 15-16 2000.
Treasury Board, 2001, p. 24.
To ensure consistency of measures, this survey could be developed in collaboration with the Institute of Aboriginal People's Health (IAPH) which plans to survey researchers and students funded through the ACADRE program annually.
See footnote 14.
A tool permitting standardized assessment across departments and institutions, such as one in use at McGill University, will be used.

Search

Funding

Institutes

Strategies

Initiatives

Signature initiatives

Strategic initiatives

Collaboration

Research highlights

Performance Measurement and Evaluation Framework

Table of Contents

Executive summary

Context and aims

Information sources

Program profile

Risk management

Evaluation issues and questions

Performance measurement and evaluation strategy

1. Introduction: context and objectives

2. Information sources and methods

3. Program profile

3.1 Overview

3.2 Program Logic Model

4. Risk management

4.1 Risk criteria

4.2 Areas of risk and mitigating strategies

5. Evaluation issues and questions

6. Performance monitoring and evaluation strategy

6.1 Evaluation design issues: comparisons and controlled studies

6.2 Overview of data sources

6.3 Ongoing performance measurement strategy

6.4 Evaluation Strategy

6.5 Reporting plan

Footnotes

Supplemental content (right column)

Strategic Training Initiative in Health Research