CIHR 5-Year Evaluation Plan 2008/09 – 2012/13

Evaluation unit
Evaluation and Analysis Branch
Knowledge Translation Portfolio
2008

Other format

Table of Contents

  1. Introduction
  2. Planning Context
  3. Approach to Developing the Evaluation Plan
  4. Evaluation Unit Activities

Appendices

1. Introduction

The Canadian Institutes of Health Research (CIHR) is in its eighth year of operation. Maturing, but still relatively young, the Agency manages high-profile health research funding programs and initiatives that are crucial to the well-being of Canadians. With expenditures of over $974M in 2007-2008 and approximately 360 employees, CIHR continues to evolve its corporate functions and services. In such a dynamic environment, the need for and the role of a strong, viable evaluation function, which can effectively assess programs' relevance, performance and contribution against CIHR's objectives and mandate,1 is increasingly important.

This document presents CIHR's first 5-year Evaluation Plan (the Plan) that has been prepared to ensure that a comprehensive, strategically-focused evaluation agenda that covers 100% of CIHR's program spending over the next five years is in place2.

The Plan was developed by the Evaluation Unit within the Evaluation and Analysis Branch between November 2007 and June 2008 and was subsequently approved by the Agency's governing bodies. It is now being implemented by the Evaluation Unit starting from the year 2008/09. The primary audiences for the Plan are CIHR senior and program management, CIHR Evaluation and Analysis Branch staff and the Treasury Board Secretariat of Canada. The Plan was developed based on previous evaluation plans, a document review, extensive consultations with management across CIHR and consideration of legislative and policy requirements for evaluation by Canada's Federal Government and Central Agencies. It covers all CIHR grant programs by proposing a strategic approach of program clusters. A risk-based prioritization strategy was used to assist in defining evaluation priorities and rationalizing the 5-year schedule. The Evaluation Unit will conduct an annual re-profiling of the 5-year evaluation schedule in order to ensure responsiveness and relevance of the 5-Year Evaluation Plan to CIHR's strategic needs, as well as to take into account the Plan's implementation feasibility. An updated evaluation schedule will be posted on CIHR's website in the first quarter of each fiscal year.

2. Planning Context

The Evaluation Plan has been developed taking into account the broader environment within which CIHR operates. As a federal organization, CIHR is subject to central requirements common to all federal institutions, in addition to its own statutory regulations.

2.1. CIHR, Government and Central Agency Priorities

2.1.1. CIHR Priorities

The CIHR Act (2000) documents the Agency's commitment to ensure transparency and accountability for the investments made in health research (s.4,l). It requires the CIHR Governing Council to monitor, analyze and evaluate issues pertaining to health and health research (s.5,d) and the overall performance of CIHR with respect to the achievement of its objectives (s.14,b). The CIHR Act also calls for a quinquennial review of the mandate and performance of every Health Research Institute (s.21).

The CIHR Policy on Evaluation and Performance Measurement, released in 2002, builds on and refers to the CIHR Act. The policy aims to promote results-based management throughout the organization and requires that each Institute, program and initiative be evaluated within five years of initiation.

CIHR's Strategic Plan, Blueprint (2004), further commits to evaluating performance and demonstrating value for money invested in health research at two different, yet complementary levels: the value of individual programs within CIHR's suite of programs and the overall return on investment to Canadians.

2.1.2. Government Priorities

In 2006, Canada's Federal Government released 'Advantage Canada: Building a Strong Economy for Canadians', a strategic, long-term economic plan designed to improve the country's economic prosperity. The document introduced five Canadian "competitive economic advantages":

  • Tax Advantage - reducing taxes for Canadians and establishing the lowest tax rate on new business investment in the G7.
  • Fiscal Advantage - eliminating Canada's total government net debt in less than a generation.
  • Entrepreneurial Advantage - reducing unnecessary regulation and red tape and increasing competition in the Canadian marketplace.
  • Knowledge Advantage - creating the best-educated, most-skilled and flexible workforce in the world.
  • Infrastructure Advantage - building the modern infrastructure Canada needs.

Among these there were initiatives aimed to ensure that existing government programs and spending were effective. 'Advantage Canada' thus introduced a new expenditure management system to be developed by the President of the Treasury Board to focus on value for money and good management.

In December 2006, specific measures to help increase transparency and strengthen accountability in government operations were brought forward by the Federal Accountability Act, which included amendments to key legislative documents such as the Financial Administration Act. Of significance, the amended Financial Administration Act (FAA) requires all grant and contribution programs be reviewed every five years for relevance and effectiveness (s.42.1) 3.

The 2007 Government of Canada's Science and Technology Strategy has also set important directions for CIHR. The strategy puts further emphasis on enhancing accountability and calls for improving our ability to measure and report on the impacts of science and technology investments.

2.1.3. Central Agency Priorities

The Treasury Board Secretariat Policy on Evaluation, released in 2001, governs the role and operation of the evaluation function across federal departments. In response to the Government of Canada's increasing emphasis on transparency and accountability in recent years, TBS has initiated a renewal of its Evaluation and other policies.

Expected to come into effect in 2008/2009, the new Evaluation Policy, Standards and Directive will broaden the scope and reshape the structure of evaluative activities in the government. The new policy suite will require CIHR to plan for 100% evaluation coverage of direct program spending over a five-year period, most of which is already covered by the FAA. With the requirement for additional evaluation coverage, the policy expands its definition of evaluation instruments and approaches in an effort to ensure the quality, neutrality and timeliness of evaluations.

Effective as of October 2008, the revised Treasury Board Policy on Transfer Payments further reinforces the requirement of the FAA that grants, contributions and other transfer payments be reviewed or evaluated every 5 years. In addition, the policy emphasizes that a performance measurement strategy is established and maintained throughout a transfer payment program's lifecycle and further clarifies that recipients of grants may be required to report on results achieved.

2.2. Risk Assessment

CIHR works in an environment of ever-evolving economic, technological, social and political conditions. In order to advance its mandate and deliver on commitments, CIHR considers various risk factors that have direct or secondary impact on its performance at the strategic, program and administrative levels.

A series of agency-specific risks, pertinent to its evaluation function, were defined in the integrated results-based management and accountability framework (RMAF) and risk-based audit framework (RBAF) for CIHR's research projects and personnel support grant program, submitted to TBS in 20064:

  • Risk of not achieving the planned outcomes -- CIHR commits to a broad set of expected results, including producing high quality research (knowledge), building health researcher capacity, contributing to the development of a robust health sector and to the health of Canadians.
  • Peer-review failure - the dramatic expansion of new initiatives, new panels and new funding streams at CIHR puts a considerable pressure on the peer review system.
  • Misuse of funds - CIHR manages a total budget of over $850M/year. Over 90% of this budget is for grants and awards that are paid to second parties -- universities and researchers -- to conduct research.
  • Getting the balance right between strategic (investments in high priority areas) and open (researcher-initiated studies in any area of health research) programs, and between disciplines.
  • Knowledge translation into results for Canadians -- 2005 evaluations of CIHR Institutes pointed to weaknesses in the area of knowledge translation.
  • Inability to adequately measure and report on the results of research - while grants are awarded based on track records of researchers and the soundness of research proposal, there is no guarantee that research will indeed be of high quality or that results will be produced at the end of the grant period.
  • Canada may lose health research talent as CIHR is not able to fund all the excellent applications that are received on an annual basis.

In addition to the above-mentioned risks, future evaluative studies and plans will also be informed by the Agency's corporate risk profile (under development). This project is led by the office of the CIHR Chief Financial Officer (Resource, Planning and Management Portfolio). To date, a complete list of corporate risks affecting CIHR has been identified and assessed on impact and vulnerability scales.

Over and above refining risk management practices at the departmental level, CIHR is also engaged in a Health Portfolio project that has developed an integrated risk management (IRM) framework for grants and contributions across the Government of Canada's Health Portfolio. The project involves CIHR, Public Health Agency of Canada and Health Canada. The IRM framework will be a step forward towards promoting holistic and systematic management of risks and ensuring greater alignment with government risk frameworks and initiatives, thus enabling strategic decision-making and results.

2.3. Evaluation and Related Functions at CIHR

CIHR has been building internal capacity for evaluation, performance measurement and analysis since 2001, largely focusing on fulfilling the mandatory requirements of the CIHR Act. In 2005 - 2007, significant investments were made in the Evaluation and Analysis Branch establishing three distinct units: Evaluation, Data and Analysis, and Impact Assessment5. The Branch was also moved to the Knowledge Translation Portfolio. The current structure of the Branch was put into place to support evidence-based decision making and strategic planning at CIHR and to satisfy the requirements of the Financial Administration Act and TBS Policies.

Presently, the Branch structure comprises of highly qualified personnel, including data specialists, evaluators and analysts of health research. It should be noted that the new Branch has yet to be fully staffed due primarily to challenges recruiting and retaining specialized staff. As well, since development of the Branch took place between 2005 - 2007 a large proportion of Branch staff are new to CIHR. Current staff are now engaged in a range of evaluative and analysis activities, such as program evaluation, research evaluation (or impact assessment) and data analyses, which all support the organization's senior management in decision-making and in meeting accountability and reporting requirements.

2.3.1. Program Evaluation and the Evaluation Unit

In the government of Canada, program evaluation is "the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results" (draft TBS Evaluation Policy, 2008). The goal of program evaluation is to inform decisions with respect to the implementation and effectiveness of a specific program or suite of programs. These evaluations can be formative or summative or have components of both. At CIHR, the program evaluation function is performed by the Evaluation Unit.

The major responsibilities of the Evaluation Unit include the following:

  • Develop annual and multi-year evaluation plans; conduct program evaluation studies in accordance with established plans and TBS requirements;
  • Lead and/or participate in inter-departmental evaluations;
  • Support program management in assessing the design, delivery and effectiveness of policies, programs and initiatives (e.g., provide advice on results-based management and accountability frameworks, the conduct of program-level evaluative studies, performance measurement activities and data analyses);
  • Engage in framework and methodology development for program-level evaluation research;
  • Support efforts to build evaluation capacity, infrastructure and culture at CIHR.

There are about 150 different funding programs at CIHR6 that are categorized in 22 program activities under three strategic outcomes: Advances in Health Knowledge, People and Research Capacity, and Knowledge Translation and Commercialization.

Apart from interest in evaluating outcomes of these programs, CIHR senior management and other stakeholders have highlighted the importance of measuring the impacts of scientific research supported by the Agency. The Impact Assessment Unit within the Evaluation and Analysis Branch is tasked with the research evaluation function.

2.3.2. Research Evaluation and the Impact Assessment Unit

Research evaluation, or impact evaluation, at CIHR aims to capture and describe the longer-term outcomes and impacts of CIHR funded health research and is not focused on a specific program. It is intended to provide information relevant for strategic planning and policy purposes, as well as to help demonstrate progress on the organization's mandate and strategic outcomes.

In 2005, CIHR recognized the need to have a framework to consider the impacts of health research. In 2007, the Impact Assessment Unit was formed and began work to finalize and implement the impact assessment framework developed in 2005. Currently, the Impact Assessment Unit is in the process of implementing this framework through several projects, including theme-based framework implementation (e.g., impacts of obesity research, impacts of cardiovascular and cerebrovascular research) and category-based framework implementation (e.g., commercialization). After this first year implementing the CIHR health research impact framework, the framework and implementation methods will be reviewed and refined as necessary.

The Impact Assessment Unit is also engaged in a number of international collaborations.

2.3.3. Data Analysis and the Data and Analysis Unit

Since the inception of the Data and Analysis Unit, the Agency's analytical capacity has been substantially strengthened. The Unit's two teams -- Data Production and Analysis -- are engaged in producing and validating CIHR statistics, responding to data requests7, preparing analyses for internal and external clients, contributing to CIHR documents/reports and providing other analytical services.

The Unit is working on the development of a new research classification system that will enable CIHR to accurately report on its investments and outputs by area of research, a research reporting system (RRS) to capture the outputs and outcomes of health research investments, and a data quality assurance framework to ensure that CIHR data are reliable and valid.

2.4. Challenges Facing Program Evaluation at CIHR

The current CIHR 5-year Evaluation Plan primarily covers program evaluations and will be implemented by the Evaluation Unit with collaboration and support from the Data and Analysis Unit as well as the Impact Assessment Unit.

In implementing CIHR's first 5-year Evaluation Plan, the Evaluation Unit may be facing a number of challenges related to financial, operational, technical and cultural factors in the organization, including:

  1. budgetary and human resources constraints;
  2. complexity of the structure of the organization, including multiple program responsibility centres and their geographical dispersion;
  3. organizational culture that has not been fully transformed to support results orientation;
  4. effective and sustainable engagement of program management in consultation and evaluative processes;
  5. lack of performance information and a data infrastructure within CIHR to support measurement and evaluation activities;
  6. in addition, there are also a number of challenges inherent to measuring program outcomes, including incrementality and attribution.

In response to some of the afore-mentioned challenges, the Evaluation and Analysis Branch has been heavily involved in a number of projects that aim to develop consistent and common performance indicators as well as related measurement capacity at CIHR, as well as initiatives that aim to enhance the overall evaluation capacity of the organization. These include:

  • Development of the Agency's Management, Resources and Results Structure (MRRS) that defines strategic outcomes, related program activities, resource requirement, along with expected results and performance indicators.
  • Ongoing development of results-based management and accountability frameworks (RMAFs) for CIHR's funding programs.
  • Implementation of the CIHR Common Framework, or common performance indicators to be collected across all 13 CIHR Institutes;
  • Development of the Research Reporting System, i.e., a centralized electronic end-of-grant data repository aimed to support capture and analysis of research outputs and impacts from researchers in a more rigorously structured and systematic manner;
  • Finalization and implementation of the CIHR Impact Assessment Framework - a framework that provides direction and guidance with respect to evaluating short and longer-term impacts of health research.
  • Design and implementation of the CIHR Evaluation Network, which is a CIHR-wide website and working group dedicated to sharing information on evaluation and performance measurement across the organization, supported by monthly face-to-face meetings.

The next section presents quality assurance practices used by the Evaluation Unit.

2.5. Quality Assurance Practices

The Evaluation Unit conducts its work in a thorough, objective, consultative manner, using a mixture of in-house and contractual resources. The following professional standards, defined by the Canadian Evaluation Society, guide CIHR's evaluation products and services: utility, feasibility, propriety and accuracy8.

Quality control of evaluation products and services in the Evaluation Unit is exercised through a number of practices, including:

  • oversight of high-profile evaluation studies by project steering committees/working groups that involve key program stakeholders and Evaluation and Analysis staff;
  • integration of multiple lines of evidence in every evaluation as a standard of good practice;
  • internal peer and management reviews;
  • existence of an FTE (Evaluation and Quality Control Officer) dedicated to quality assurance;
  • review of contractor proposals for their methodological rigor and quality assurance practices.

The governance of evaluation at CIHR is supported by several governance and strategic management committees and subcommittees, which include experts in evaluation and performance measurement. These advisory bodies enhance the quality assurance system of the program evaluation function through a collegial and consultative approach to decision-making and approval. Feedback from CIHR's committees and senior management confirms that the Unit is delivering increasingly comprehensive, credible results and services.

3. Approach to Developing the Evaluation Plan

3.1. Guiding Principles

A number of guiding principles, informed by the TBS evaluation standards and guidelines, were taken into consideration while developing the Plan:

  • Using the planning process as an opportunity to communicate the role and importance of evaluation and engage CIHR management in evaluation prioritization and use;
  • Adopting a risk-based approach to identify evaluation needs and focus earlier on important priorities and high-risk projects;
  • Utilizing a consistent, balanced and transparent process to determine evaluation commitments;
  • Aligning scheduled evaluations with the CIHR Management, Resource and Results Structure;
  • Building in sufficient flexibility to assure that the Agency and the Evaluation Unit are able to respond to emerging, high-priority evaluation needs;
  • Aligning the Plan with the planning of the International Review of CIHR as an opportunity to continue to strengthen the role of evaluation as an effective and essential management function9;
  • Incorporating CIHR's commitment to meet the requirements of the Financial Administration Act and the revised TBS Evaluation Policy and Policy on Transfer Payments.

These principles reflect the importance for CIHR to ensure that evaluation efforts balance accountability with program improvement and learning.

3.2. Planning Process and Important Considerations

The planning process included the following major steps: establishment of the 5-Year Evaluation Planning Working Group; review of CIHR and government-wide priorities; consultations with CIHR management at the corporate and programmatic levels; refinement of the evaluation universe by Program Activity Architecture categories; development of the risk-based prioritization methodology and prioritization/ranking of program evaluations (for description of all steps, please see Section 3.3 and Appendix C).

Additionally, the Evaluation Unit continues to monitor factors that may reshape CIHR's suite of programs and budgetary allocations and thus impact program evaluation priorities and/or foci of evaluative studies proposed in this Plan. These factors include:

  • the results of TBS Strategic Review of CIHR (conducted in spring/summer 2008),
  • the development of the new CIHR Strategic Plan (Blueprint II),
  • the completion of the Agency's corporate risk profile,
  • the change in leadership in fiscal year 2008/09, and
  • influences due to shifts in government priorities or the health needs of Canadians.

3.3. Evaluation Planning Steps

Step 1: Review of Priorities
A review was carried out by the Evaluation Unit to link prospective evaluation activities to department- and government-wide objectives and priorities. Major sources of information on departmental goals and risks were the Agency's strategic directions and priorities documents (Section 2.1). Also, in January 2008, members of the Subcommittee on Performance Measurement (SPM) were proactively approached to identify CIHR priorities and inform of forthcoming expenditure decisions that would need to be informed by evaluation. Government-wide priorities and objectives were confirmed through a review of relevant legislative documents and guidelines.

Step 2: Refinement of Evaluation Universe
Considerately attention was devoted to refining and operationalizing the evaluation universe to ensure it reflects CIHR programs well and provides full evaluation coverage of program spending. The Program Activity Architecture (PAA) was, therefore, used as a useful starting point for defining program evaluation universe. The use of the PAA structure would also enable the Evaluation Unit to study and conclude on the effectiveness of entire program areas using an optimized number of studies focused on a single broader issue.

Step 3: Selection of Risk and Priority Criteria
Given the number of evaluations required, the prioritization method and especially the selection of informative and valid risk-based criteria were important. An initial identification of priority criteria was guided by CIHR's 2005/06 evaluation planning, albeit a few modifications were introduced. As a result, one level of criteria was established: materiality, program lifecycle, accountability history, strategic/corporate priorities, and visibility (for definitions, please refer to Appendix C). The criteria selected were informed by the review of priorities and consequently validated by the 5-Year Evaluation Plan Working Group and the following committees: SPM, Research and Knowledge Translation Committee (RKTC), and the Standing Committee on Performance Measurement, Evaluation and Audit (SCPMEA).

Step 4: Prioritization of Evaluation Universe
Well-defined PAA evaluation activities were prioritized according to the established criteria. The rating scale for each criterion was based on a review of similar rating scales across federal departments and finalized in consultation with the Working Group. Based on the overall score, a prioritized list of program evaluation projects was established where the highest scoring PAA category slated for an evaluation earlier in the 5-year cycle and the lowest, later (with an exception of TB requirements & shared programs). Horizontal issues were not prioritized, but rather identified as preliminary foci for prioritized evaluations. The final product was the proposed 5-year schedule of program evaluations (Table 1).

Step 5: Validation of Prioritized Evaluation Activities
In order to strengthen the validity of the prioritization process, interviews with senior management and selected departmental directors were conducted and a qualitative analysis of their responses was performed. The analysis of interview data revealed a consensus regarding major evaluation-related priorities across CIHR, thus supporting the results of the risk-based prioritization exercise.

Step 6: Approval Processes and Dissemination
Once the finalized 5-year schedule of evaluative activities had been accepted by CIHR's decision-making bodies (SPM, RKTC, and SCPMEA), a draft 5-Year Evaluation Plan was prepared and then approved by the Agency Head of Evaluation. Upon acceptance by TBS, the Plan will be widely distributed across CIHR.

Step 7: Implementation and Monitoring
The Evaluation Unit continues to define the Plan's implementation strategy. As outlined in Section 3.2, a number of factors may impact program evaluation priorities and/or foci of evaluative studies proposed in this Plan. In response, the Evaluation Unit commits to conducting an annual re-profiling of the 5-year evaluation schedule, based on a similar type of methodology, to ensure its ongoing relevance and accuracy. The updated schedule will be widely distributed across CIHR and posted on its website.

  1. The mandate of CIHR is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system (Bill C-13, 2000).
  2. The 5-Year Evaluation Plan covers program evaluations only. Other evaluative activities, such as research impact assessments, have been excluded from this planning document (for more detail, please refer to Section 2.3).
  3. CIHR is a Departmental Corporation listed in Schedule II of the Financial Administration Act. Furthermore, CIHR’s program spending is almost exclusively grants and awards.
  4. The methodology used for the risk assessment included a review of risk areas identified in the internal audit plans and 2005 international review of CIHR; followed by an analysis of the risk level and identification of areas of concern and impact related to the risk areas. Subsequently there was an analysis of existing mitigation strategies. Management representatives from CIHR actively collaborated in the development of the RMAF/RBAF and the risk assessment.
  5. The mandate of the Evaluation and Analysis Branch is to support CIHR leadership and management in implementing sound performance measurement, evaluation and analysis practices to help achieve CIHR’s mandate, strategic objectives and to meet the organization’s legislative requirements.
  6. As classified by the CIHR financial codes for Fiscal Year 2007/08.
  7. To submit a data request, please complete the Online Data Request Form
  8. Description of Program Evaluation Standards is available online
  9. The previous review of CIHR, conducted in 2005/06, was a large-scale international review led by Evaluation and Analysis to assess the progress of the Agency as a whole since its inception. The second review is planned for 2010/11.

Table 1.

CIHR 5-Year Evaluation Schedule (2008/09 – 2012/13)