CIHR 5-Year Evaluation Plan 2008/09 – 2012/13

[ Table of Contents ]

4. Evaluation Unit Activities

4.1. Evaluation Activities for 2008/09 – 2012/13

Fiscal Year 2007/08 has been the first year for the Evaluation Unit operating at almost full staffing capacity. In that year, a number of small and large-scale evaluation projects have been undertaken.

The current year, 2008/09, is devoted to beginning the process of fulfilling the CIHR 5-Year Evaluation Plan requirements. The evaluation commitments for 2008/2009 reflect high priority program evaluations, previously identified by CIHR management, such as the Open Operating Grants program (OGP) evaluation, evaluation of CIHR team-type programs, Randomized Control Trials, as well as TBS requirements (Pandemic Preparedness Research Initiative) and Tri-agency evaluations (Indirect Costs, Canada Research Chairs program). Finally, several evaluation studies were carried over from 2007/08 to be completed this fiscal year, such as the evaluation of the Strategic Training Initiative in Health Research (STIHR), Canada Graduate Scholarships Program (CGS), Doctoral Research Awards (DRA) and the evaluation of the Interagency Advisory Panel and Secretariat on Research Ethics (PRE/SRE).

Table 2 below documents the Agency's planned program evaluations for the upcoming 5 years (2008/09 – 2012/13). It builds and expands upon the results of the prioritization exercise. Where appropriate, each commitment has been operationally defined in terms of specific projects, approaches and resource needs.

In some instances, the evaluation information needs of managers have necessitated the re-clustering of programs within and between PAA categories. For example, the Team Grant Program PAA category has been expanded to include all 10 CIHR team-type programs, historical and existing, to ensure that management will get sufficient evidence as to the value-added of the 'team approach' to funding and conducting research. In order to evaluate the OGP, largest CIHR program, it is broken down into multiple evaluative studies, spread across several years, thereby making the evaluation more manageable and focused.

Other adjustments have included the merging of pairs of PAA categories into single evaluative studies (e.g., Training Programs and Strategic Training Programs; Salary Programs and Strategic Salary Programs), which will enable an analysis of, for example, the effectiveness of a strategic versus an open approach to funding, among others.

All evaluations included in the 5-year schedule will address value for money by covering core issues proposed by the TBS Directive on the Evaluation Function. The actual scope of projects for years 2009/10 to 2012/13 will be further defined by resources available, the status of previous evaluation projects and information needs of CIHR management following an annual re-profiling of the Plan by the Evaluation Unit.

Table 2. Proposed Evaluation Projects for 2008/09 – 2012/13.

[Please note that more precise information is known for projects slated for an evaluation in the current year than future years. Evaluations not led by the CIHR's Evaluation Unit are highlighted in grey].

PAA Sub-Activity Category

Basic stats 2007/08

MRRS Outcomes and Expected Results (CIHR MRRS, 2007)

Proposed Evaluation(s)
(if known)

Preliminary Evaluation Foci

Start
Year

Rationale / Resources / Comments

1.1.1. Open Operating Grant Program

3898 grants funded

$383M

$98.3K/ per grant

Expected Results:

  • Health research advances knowledge and leads to broader impacts such as informing decisions, impacts on health and the economy
  • Appropriate resources available that enable world class, investigator formulated research across all areas of health research

Output: Research grants

1. Peer review evaluation

Peer Review

2008/09

Given its magnitude, size and dated evaluative information available to date, this program requires comprehensive assessment that will be conducted over several years to capture evaluation issues best. Several evaluative studies are being proposed and various issues of interest will be explored (e.g., assessing the peer review process especially in the context capturing research excellence; the program’s scale and scope; international competitiveness and other issues).

The evaluations are to be conducted internally with some tasks contracted out. The following budgetary allocations have been proposed: $25,000 for 2008/09; $40,000 for 2009/10 and $35,000 for 2010/11.

2. Historical outcomes and impacts assessment

Health research impacts

2008/09

3. Program’s meta-evaluation

Overall effectiveness and relevance

2010/11

1.1.2. Randomized Control Trials Program

74 grants funded

$32.3M

$436.7K/ per grant

Expected Results:

  • Better health research, research training and mentoring through funding of excellent teams
  • Appropriate resources available that enable team grants to enhance research and training

Output: Research grants

Program’s effectiveness (analysis of the final reports data)

Health research impacts

2008/09

The RCT evaluation was identified as a high priority as a result of multiple factors: the program’s high profile on the international arena, large per grant spending ratio, the decision to merge RCT into OGP, and the immediate availability of data collected through final reports (Consolidated Standards of Reporting Trials-CONSORT).

This evaluation is to be conducted internally. The framework is now being developed in order to ensure that information needs of senior management are adequately met and sufficient evidence about the program’s outcomes and effectiveness is generated.

1.1.3. Team Grant Program

109 grants funded

$26.7M

$244.7K/ grant

Expected Results:

  • Better health research, research training and mentoring through funding of excellent teams
  • Appropriate resources available that enable team grants to enhance research and training

Output: Research grants

Evaluation of all team-type programs at CIHR:

- CAHR
- CGSP
- Groups
- Teams
- ET
- ICE
- IHRT
- NET
- PHSI
- RRG

(For Acronyms, refer to Appendix A)

Team
Research Productivity

Capacity development

Knowledge translation

2008/09

The evaluation strategy was developed by the Evaluation Unit in the Spring of 2008. The proposed evaluation will include 10 team-type programs (ongoing and historical) that are spread out across different PAA categories. Such an approach will enable evaluators to provide solid evidence on the value-added of the team approach to funding research and, potentially, enrich the theory on team performance and productivity. Special attention will be devoted to developing evaluative tools for such horizontal issues as capacity development and knowledge translation.

The project’s budget includes $25,000 in 2008/09.

2.1.3. Canada Research Chairs Program

693 grants funded

$86.4M

Expected Result:

  • World-class research capacity is enhanced in Canadian universities, research institutes and hospitals through the attraction and retention of excellent researchers

Output: Research chairholder is established

TBD
externally

Capacity Development

2008/09

External evaluation (Tri-Agency)
Lead: CRC Secretariat (SSHRC)

The evaluation is a requirement for the 2009/10 year. The planning will start in November 2008. The CRC program is intended to attract and retain some of the world's most accomplished and promising researchers; therefore, it will be of interest to CIHR to consider CRC outcomes in the context of other capacity development programs.

1.2.4. Pandemic Preparedness Research Initiative

40 grants funded

$2.8M

$70.5K/ grant

Expected Results:

  • Health research advances knowledge and responds to objectives of Pandemic Preparedness Strategic Research Initiative
  • Appropriate resources available that enable world-class research related to Pandemic Preparedness

Output: Research grants

TBD
externally

National Health Priorities

2008/09

External evaluation
Lead: Institute of Infection and Immunity

A formative evaluation will be carried out in 2008/09. The proposed evaluation will look at the relevance and effectiveness of the Initiative’s task group, researcher and stakeholder perceptions on initiative design and implementation, and success of activities aimed at fostering linkages and exchanges.

n/a
Indirect Costs Program

n/a

n/a

TBD Externally

2008/09

External evaluation
Lead: CRC Secretariat (SSHRC)

A comprehensive evaluation of the program to be carried out during its sixth year of operation is a TB requirement. The primary focus of the evaluation is the continued relevance of the program and its cost-effectiveness as well as its results, expected outcomes and impacts. The project is currently scheduled to start in September 2008. CIHR will be participating in a working group.

1.2.2. Large Strategic Initiatives Programs

172
grants funded

$20.8M

$121.2K/ grant

Expected Results:

  • Health research advances knowledge and leads to broader impacts such as informing decisions, impacts on health and the economy and responds to objectives of large, cross-cutting strategic initiatives
  • Appropriate resources available that enable world-class research responding to cross-cutting strategic health priorities

Output: Research grants

Cluster evaluation of large strategic initiatives

Health Research Priorities

Health Research Impacts

2009/10

Evaluating large, cross-cutting strategic initiatives is important from the decision-making perspective. The proposed evaluation will look at the model of funding strategic initiatives holistically (rather than success of each individual initiative) and will build upon the previous Obesity Initiative evaluation framework and impact assessment study, as well as others. It will be conducted internally by the Evaluation Unit.

2.2.1. Research Resources and Collaborations

420
grants funded

$49.4M

Expected Results:

  • High quality research resources are available for excellent health research as a result of effective funding programs
  • Appropriate resources available that enable adequate research resources for health research

Output: Research grants

TBD

Partnerships
Citizen engagement

2009/10

The proposed timing for this evaluation may need to be re-examined on the materiality and overall scores, given that the team-type programs that constitutes 2/3 of the budget for this category are evaluated with the Team Grant PAA.

The scope of the evaluation will be determined through future consultations.

2.1.1. Salary Support Programs

416
awards funded

$21.1M

$50.8K/ award

Expected Results:

  • Medical/Health, University and Research Institution staff have dedicated time for health research
  • Appropriate resources available that enable world-class health researchers across all areas of health research to devote more time to health research responding to research opportunities

Output: Salary Awards and research allowances

Programs’ relevance and effectiveness

Capacity Development

2009/10

In 2004/05, the Task Force on Career Support examined CIHR’s suite of programs offering salary awards to health researchers and made a series of recommendations. It is expected that a more comprehensive evaluation of salary programs and their role in enabling robust research environment will be performed in 2009/10 by the Evaluation Unit.

2.1.5. Strategic Salary Support Programs

50
awards funded

$2.7M

$54.8K/ award

Expected Results:

  • Medical/Health, University and Research Institution staff have dedicated time for health research on strategic initiatives
  • Appropriate resources available that enable world-class health researchers to devote more time to health research in strategic areas of health research

Output: Salary Awards and research allowances

n/a
External Review of CIHR

n/a

n/a

TBD

2010/11

As required by the CIHR Act (2000), the Agency should conduct a quenquennial review of CIHR’s Research Institutes. The Evaluation and Analysis Branch will take the lead for this high profile project which will include a review of the slate of Institutes.

1.2.1. Strategic Priority Operating Grant Program

430
grants funded

$38.2M

$88.9K/ grant

Expected Results:

  • Health research advances knowledge and leads to broader impacts such as informing decisions, impacts on health and the economy and responds to objectives of strategic initiatives
  • Appropriate resources available that enable world-class Institute-framed research responding to strategic health priorities

Output: Research grants

TBD

Health Research Priorities

2010/11

The scope and methodology for evaluating the strategic priority operating grant program will be developed in conjunction with the evaluation of the large strategic initiatives and the open operating grants program.

1.2.5. Expensive Drugs for Rare Diseases Initiative

1 grant funded

$11.6M

$11,600K/ grant

Expected Results:

  • Additional information available regarding the use of enzyme replacement therapies to treat patients suffering from Fabry Disease

Output: Research grant

TBD
externally

National Health Priorities

2010/11

External evaluation
Lead: Scientific Oversight Committee

This initiative is a joint investment by the Government of Canada, participating provinces and the private sector, with the objective of gathering therapeutic effectiveness information on the rare genetic Fabry Disease. The Independent Scientific Oversight Committee is responsible for monitoring, evaluating and communicating publicly the results of the initiative’s research.

2.3.2. Partnership Programs

263
grants funded

$9.1M

$34.7K/ grant

Expected Results:

  • Relevance and quantity of health research increased through innovative national and international alliances
  • Appropriate resources available that enable partnered health research

Outputs: Research grant; Training Awards; Salary Awards

Cluster evaluation of partnership programs

Partnerships

Knowledge Translation

2010/11

The Partnership program provides grants to enable national/international programs to coordinate health research activities with stakeholders. Given similar goals/scope of the program with the KT program, the feasibility of a joint evaluation will be further examined.

3.1.1. Knowledge Translation Funding Program

192
grants funded

$9.5M

$49.5K/ grant

Expected Results:

  • Health research is translated more effectively as result of funding programs
  • Appropriate resources available that enable world-class knowledge translation of health research

Output: Research grants

Knowledge Translation Study

Knowledge
Translation

Partnerships

2010/11

Evaluation of two types of KT (end-of-grant and integrated KT) will be at the core of the proposed evaluative study. CIHR Knowledge Exchange and Synthesis Directorate will be involved.

3.2.1. Commercialization Funding Programs

490
grants funded

$98.9M

$201.9K/ grant

Expected Results:

  • Health research is commercialized more effectively as a result of funding programs
  • Appropriate resources available that enable commercialization of health research

Output: Research grants

Outcomes and impacts of commercialization of health research

Knowledge Translation

Partnerships

Health Priorities

2010/11

An initial evaluability assessment is to be carried out, followed by development of the performance measurement and evaluation framework ($35,000 budget for 2009/10). Future evaluative activities are contingent upon the framework.

1.2.3. HIV/AIDS Research Initiative

199
grants funded

$17.3M

$87.1K/ grant

Expected Results:

  • Health research advances knowledge and responds to objectives of HIV/AIDS Research Initiative
  • Appropriate resources available that enable world-class research related to HIV/AIDs

Output: Research grants

TBD
externally

National Health Priorities

2010/11

External evaluation
Lead: PHAC

4 federal departments are responsible for delivering this Federal Initiative, launched in 2004/05 (renewal in 2010). The Public Health Agency of Canada will lead the evaluation.

2.1.2. Training Support Program

796
awards funded

$24.2M

$30.5K/ award

Expected Results:

  • Highly trained individuals available for health research
  • Appropriate resources available that enable superior health research trainees in all areas of health research to be qualified to pursue careers in health research

Output: Training Awards and research allowances

Cluster evaluation of all training programs

Capacity Development

2011/12

Training programs have been well evaluated in 2006-2008 as individual programs. Building upon this work, an over-arching evaluation that looks at the effectiveness of all training programs (incl. capturing excellence) is proposed.

2.1.6. Strategic Training Support Program

299
awards funded

36.3M

$121.6K/ award

Expected Results:

  • Highly trained individuals available for health research in strategic priority areas of health research
  • Appropriate resources available that enable superior health research trainees to pursue careers in strategic areas of health research

Output: Training Awards and research allowances

1.2.6. National Anti-Drug Strategy Treatment

0 grants funded

Expected Results:

  • Appropriate resources available that enable world-class research related to improving the effectiveness and informing policy and program development of and for addiction treatments (for youth, First Nations and Inuit peoples)

Output: Research grants

TBD
externally

National Health Priorities

2011/12

External evaluation
Lead: Health Canada

This is a new initiative with first grant programs launched in early 2008. Part of Government’s National Anti-Drug Strategy, the evaluation will be led by Health Canada.

2.1.4. Canada Graduate Scholarship Programs

747
awards funded

$13.9M

$18.6K/ award

Expected Results:

  • Increased capacity to meet demand for highly qualified personnel in the academic, public and private sectors
  • Appropriate resources available that enable superior health research trainees in all areas of health research to be qualified to pursue careers in health research

Outputs: Training Awards

TBD

Capacity Development

2011/12

Tri-Agency
Lead: CIHR

This Tri-Agency evaluation will be led by CIHR and its Evaluation Unit. It will build upon the 2007/08 study in conjunction with the evaluation of CIHR’s training programs.

3.1.2. Networks of Centres of Excellence Grants

11 grants funded

$27.5M

Expected Results:

  • Strong linkages and partnerships created between university, government and industry, and other users (NGOs), resulting in the transfer of knowledge and exploitation of leading-edge research results with economic or societal benefits to Canada

Output: Networks supported

TBD
externally

Capacity Development

Partnerships

2011/12

External evaluation (Tri-Agency)
Lead: NSERC

The evaluation will build upon the 2006 study. It will be led by the Natural Sciences and Engineering Research Council of Canada.

2.3.1. Institute Support Grant

14
grants funded

$13M

Expected Results:

  • Effective domestic and international health research agendas as well as alliances and partnerships in areas related to the mandate of each Institute

Outputs:

  • Discussion documents and other publications
  • Communications regarding funding opportunities and funding results
  • Relationships and networks
  • Established Strategic Priorities

TBD

TBD

2012/13

Institute support grants are being redesigned following extensive consultations to address TB concerns. The timing of this evaluation would allow to prepare for renewal of the terms and conditions of this grant in 2013/14.

2.4. Ethical, Legal, Social Issues

50
grants funded

$1.5M

$30.8K/ grant

Expected Results:

  1. Health research conducted more ethically as a result of effective funding programs

Output: Research grants

TBD

Ethics

2012/13

This evaluation is to be carried out internally, building upon the 2008/09 PRE/SRE study.

4.2. Performance Indicators

As part of the evolving performance measurement system at CIHR, the Evaluation Unit will report on its performance using a small set of key performance indicators:

  • Percentage of annual work plan and 5-Year Evaluation Plan accomplished. This could be a measure of the Unit's ability to adhere to project schedules and to efficiently manage the evaluation process.
  • Quality and usefulness of products and services, as assessed by clients. This could be accomplished via a combination of post-evaluation project client surveys, together with an annual survey of all senior managers and key stakeholders.

4.3. Resource Requirements

The Evaluation Unit has a resource base of 8 FTEs. The Unit's staff consists of the following personnel: one manager, one evaluation and quality control officer, 4 evaluators10 and 2 junior evaluators. One additional evaluator position is being proposed for the coming fiscal year.

It is expected that much of the evaluation work will be conducted using in-house resources.

In addition to undertaking program evaluation projects as defined in previous and current evaluation plans, the Evaluation Unit dedicates considerable amount of time to providing support and advice to other organizational functions. Approximately 20 to 30% of evaluators' time is devoted to client support. The Unit supports program management in assessing the design, delivery and effectiveness of policies, programs and initiatives (e.g., advice on results-based management and accountability frameworks, conduct of program-level evaluative studies, performance measurement activities and data analyses).

If you require more information about client support and for contact information, please refer to the next section.

4.4. Evaluation and Analysis Contacts

Anyone requiring the assistance of the Evaluation and Analysis Branch or needing more information should contact:

Peggy Borbey, Director
Evaluation and Analysis Branch
Knowledge Translation Portfolio
Telephone: 613-941-4350
E-mail: peggy.borbey@cihr-irsc.gc.ca

For more information about this Evaluation Plan, program evaluation projects or client support, contact:

Laura McAuley, Manager
Evaluation Unit / Impact Assessment Unit
Evaluation and Analysis Branch
Knowledge Translation Portfolio
Telephone: 613-948-2283
E-mail: laura.mcauley@cihr-irsc.gc.ca

We will be pleased to offer our support and provide you with the information you need.

Your feedback and comments are welcome.

10. One evaluator is currently on maternity leave. She will return to her duties in March 2009.