Canadian Institutes of Health Research Evaluation Plan 2010/11 - 2014/15
2. Evaluation Schedule
This section on evaluation scheduling has been divided into three parts:
- Identifying the evaluation universe and prioritizing evaluations according to need and risk;
- The scheduling and resourcing plan for 2010/11 including resource utilization for evaluations and other core Evaluation Unit activities;
- The five-year evaluation and resource schedule from 2010/11 to 2014/15 and summary of evaluations to be conducted during this period.
2.1 Evaluation Universe and Prioritization
CIHR's identified program universe of direct program spending is based on its 2010/11 PAA and Management Resources and Results Structure (MRRS), with activities identified at the sub- and sub-sub activity levels. This program universe includes activities which have TB submissions specifying the program lifecycle and when the program should be evaluated. Other activities, such as the Open Operating Grants Program, do not have a defined lifecycle or evaluation timeline, aside from overall TB Policy requirements to evaluate all program spending within a five-year period.
The factors used to determine the scheduling of evaluations between 2010/11 and 2014/15 therefore include:
- Defined evaluation timelines — drawn from program terms and conditions, performance measurement strategies or RMAFs submitted to TBS. Many horizontal evaluations with NSERC and SSHRC fall into this category and have pre-determined evaluation schedules;
- Providing evidence for program redesign — CIHR is currently redesigning a range of its open and strategic programs. Evaluations have been scheduled to maximize the opportunity to inform program redesign;
- Risk-based assessment — prioritization of evaluations based on factors such as materiality, program lifecycle and visibility of the program. This year's assessment is based on a refresh of a full risk-based assessment exercise conducted at the time of CIHR's first EP (2008), and is updated based on changes to PAA categories, consultations with senior staff within CIHR and materiality;
- Timing evaluations to maximize efficiencies — for example, clustering programs with similar objectives or harmonizing evaluation timelines with CIHR's International Review process.
More detailed descriptions of rationale for the scheduling of evaluations can be found in Table 5 provided in Section 2.3.
2.2 Scheduling and Resourcing Plan for 2010/11
The following schedule outlines the timing of program evaluations at CIHR for Fiscal Year 2010/11. Timelines for completion of CIHR-led evaluations are in-line with those specified in Tri-Agency evaluation planning. Typically, evaluations require between three and six months for planning and between nine months to one year for the conduct of the study, at the end of which an evaluation report is produced. These are estimates based on the current resources available to the Evaluation Unit and the governance structures in place at CIHR. As part of its continuous improvement process, the Evaluation Unit will be exploring options for making its processes and products more useful and timely for management.
Scheduling and Resource Plan for 2010/2011
Planning an Evaluation
CIHR Programs
1.1.1 - Open Operating Grant Program (Q4)
1.4.1.1 - Strategic Initiatives Programs Including Regenerative Medicine & Nanomedicine Initiative & Canadian Light Source Program (Q1)
1.4.2 - Partnerships for Health System Improvement ((PHSI & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Q3)
Tri-Agency Programs and Horizontal Initiatives
1.3.2.3 - Centres of Excellence for Commercialization and Research Program (Q3)
1.4.1.5 - National Anti-Drug Strategy (NADS) Treatment Research Initiative (Q3)
Ongoing Evaluation
CIHR Programs
1.1.1 - Open Operating Grant Programs (Q1, Q2)
1.1.2 - Randomized Controlled Trials Program (Q1, Q2, Q3)
1.2.1.1 - Salary Support Programs (Q1, Q2, Q3)
1.4.2.1 - Strategic Initiatives Programs Including Regenerative Medicine & Nanomedicine Initiative & Canadian Light Source Program (Q2, Q3, Q4)
1.4.3 - Partnerships for Health System Improvement ((PHSI & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Q4)
Tri-Agency Programs and Horizontal Initiatives
2.1.3 - Canada Research Chairs Program (Q1, Q2, Q3)
1.3.2.2 - Business-Led Centres of Excellence (Q3, Q4)
1.3.2.3 - Centres of Excellence for Commercialization and Research Program (Q4)
1.4.1.5 - National Anti-Drug Strategy (NADS) Treatment Research Initiative (Q4)
Evaluation Report
CIHR Programs
1.1.3 Open Operating Grant Programs (Q3)
1.1.4 Randomized Controlled Trials Program (Q4)
1.2.1.1 Salary Support Programs (Q4)
Tri-Agency Programs and Horizontal Initiatives
2.1.3 Canada Research Chairs Program (Q4)
Resource planning assumptions to deliver the 2010/11 schedule are based on five evaluator FTEs, two junior evaluator FTEs and one evaluation manager FTE. Resource requirement calculations are based on 12 person months per FTE to a total of 96 person months for the Evaluation Unit as a whole for the fiscal year 2010/1112. As of August, 2010 the Evaluation Unit had six FTEs including the evaluation manager, with two vacant FTEs that are in the process of being filled. Furthermore, as there are only seven months left in this fiscal year, the FTE was recalculated to reflect the remaining amount of months for this year. The pro-rated FTE is 56 person months for the Unit as a whole.
The Evaluation Unit's annual salary and operating budget for 2010/11 are as follows:
| Salary (based on full staffing of eight FTEs): | $576K |
| Contracting: | $55K |
| Core budget (inc. training): | $14K |
| Total: | $645K |
These resources are adequate to deliver the 2010/11 plan. However, the required budget for contracting is anticipated to increase in 2011/12 as CIHR will be leading on the combined Vanier and Canada Graduate Scholarship Program Evaluations on behalf of the Tri-Agencies. A combined contracting budget of $240K is assumed to conduct this evaluation in 2011/12, with a CIHR contribution of $70-$80K. This will be in addition to other contracting requirements in 2011/12 of $50K for surveys, bibilometric studies or other contracted elements of in-house evaluations.
Under its current resourcing model, CIHR is committed to conducting evaluations in-house where possible and does not typically contract full evaluations 'end to end.' An estimated equivalent contracting cost for each evaluation that is planned to be conducted in-house would be around $100K-$200K per evaluation, or $400K-$800K for 2010/11. Both NSERC and SSHRC contract the majority of their evaluations, particularly for larger programs. The TBS Centre for Excellence in Evaluation recently indicated13 that there is an intent to reduce the number of evaluations that are contracted externally by departments and agencies and to conduct a greater number in-house. In this respect, CIHR's approach to evaluation will be in-line with future TBS direction in this area.
Resource utilization by the Unit can be divided into the following core activities:
- In-house evaluations — these relate to evaluations led and conducted by CIHR Evaluation Unit staff, sometimes including a small contracting budget (generally around $25K) for specialized services such as surveys or bibliometric studies.
- Tri-Agency/Horizontal evaluations — these relate to horizontal evaluations, frequently involving SSHRC and NSERC, which are fully contracted to consultants. Where CIHR is responsible for leading an evaluation for the Tri-Agencies, the FTE requirement is greater as Evaluation Unit staff lead on working closely with the contractor in the design, conduct and communication of the evaluation. Where another agency takes the lead, FTE requirements relate to Unit staff participation in working groups, reviewing all evaluation outputs such as frameworks, research instruments, technical and evaluation reports, making data requests and leading on internal communication of the evaluation within CIHR.
- Evaluation and performance measurement advice and guidance — this relates to the Evaluation Unit's role in acting as an internal resource for CIHR and the Institutes in providing advice on the design and conduct of evaluations. This can take a range of forms, including assistance with designing measurable program outcomes with program managers to reviewing and advising on Institute-led evaluations.
- Implementation of Treasury Board Evaluation Policy Suite — ongoing work to ensure and maintain full compliance with the Treasury Board's new Policy, Directive and Standard on Evaluation. This includes annual production of the Evaluation Plan and related consultations and risk-based analyses, assisting relevant committees (EMC, SPM, Audit Committee) in their roles and responsibilities in relation to the plan and liaising with TBS as required.
- Building and maintenance of evaluation capacity and capability — staff training in evaluation methodologies, attendance and participation in Canadian Evaluation Society National Capital Chapter (CES) meetings, briefings and learning events, obtaining the new CES Credentialed Evaluator professional designation, and participation in CIHR organized training programs.
- Involvement in corporate initiatives and representing the Evaluation Unit within CIHR and externally — this includes involvement within CIHR in working groups on Roadmap Implementation, Common CV Renewal, corporate data planning, PAA redesign or other such groups to ensure the Unit is adding value through its expertise and having its requirements considered. This could also include special areas of focus as requested by the Evaluation Committee (EMC), such as evaluating CIHR's relationships with institutions or its peer review processes. Externally, this would include regular meetings of the Interagency Evaluation Steering Committee and a cross-departmental working group on evaluating horizontal initiatives.
The resource plans for program evaluations and other evaluation activities in 2010/11 are presented in Tables 3 and 4.
Table 3 - Resource Planning 2010/11 (Evaluations)
Total available FTE resources (once fully staffed) = 96 months for full fiscal year and 56 months pro-rated for remaining seven months of fiscal year 2010/11.
EM = Evaluation Manager
E = Evaluators
JE = Junior Evaluators
| Evaluation | PAA | Evaluation type | CIHR Evaluation Unit Activities 2010-2011 | Person months - 2010-11 | Contract elements | CIHR Contract $ 2010-11 | Funding source 2010-11 |
|---|---|---|---|---|---|---|---|
| Open Operating Grant Program - Relevancy and Effectiveness evaluation | 1.1.1 | CIHR – In-house |
Design of evaluation (report to be completed in 2011/12) | EM - 0.5 E - 1 JE - 1 Total - 2.5 |
None | $0K | n/a |
| Salary Support Programs | 1.2.1 | CIHR – In-house |
Design, conduct and report on evaluation | EM - 1.5 E - 8.5 JE - 3.5 Total - 13.5 |
Survey of researchers | $25K | Evaluation Unit |
| Canada Research Chairs | 1.2.1.3 | Tri-Agency- SSHRC lead | Review and approve technical reports, evaluation reports, attend working groups | EM - 0.5 E - 3 JE - 2.5 Total - 6 |
Fully contracted by SSHRC | $0 ($288K via SSHRC) |
Secretariat |
| NCE/BL-NCE | 1.3.2.2 | Tri-Agency-NSERC lead | Review and approve technical reports, evaluation reports, attend working groups | EM - 0.25 E - 1 JE - 0.75 Total - 2 |
Fully contracted by NSERC | $0 ($250K via NSERC) |
Secretariat |
| CECR | 1.3.2.3 | Tri-Agency-NSERC lead | Review and approve technical reports, evaluation reports, attend working groups | EM - 0.25 E - 0.5 JE - 0.25 Total - 1 |
None | $0 ($150K via NSERC) |
Secretariat |
| Strategic Initiatives (including Regenerative Medicine and Nanomedicine and the Canadian Light Source Program) |
1.4.1.1 | CIHR – In-house |
Design and conduct part of the evaluation | EM - 1 E - 8 JE - 3 Total - 12 |
None | $0 | n/a |
| National Anti-Drug Strategy (NADS) Treatment Research Initiative | 1.4.1.5 | Horizontal- Justice Canada | Review and approve technical reports, evaluation reports, attend working groups | EM - 0.25 E - 0.5 JE - 0.25 Total - 1 |
TBD | $0 | Justice Canada |
| Partnerships for Health System Improvement (PHSI) & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Knowledge Translation Programs) | 1.4.2 | CIHR – In-house |
Design and conduct part of the evaluation | EM - 1 E - 3 JE - 2 Total - 6 |
TBD | $30K | Evaluation Unit |
| Finalizing reporting for two evaluations (OGP; RCT) | 1.1.1 &; 1.1.2; | CIHR – In-house |
Drafting and finalizing reports for two completed evaluation and one evaluative study. Working with management on response, submitting for approval, submitting to TBS | EM - 0.5 E - 4.5 JE - 2 Total - 7 |
None | $0 | n/a |
| Total | - | - | - | 51 mths14 | - | $50K | - |
Table 4 - Resource Planning 2010/11 (Other Evaluation Unit activities)
| Activity | Person months - 2010/11 | Contracting/training $ |
|---|---|---|
| Evaluation and performance measurement advice/guidance for CIHR and the Institutes | EM - 0.75 E - 4 Total - 4.75 |
$0 |
| Implementation of TBS Policy Suite | EM - 0.5 E - 1 Total - 1.5 |
$0 |
| Build and maintain evaluation capacity | EM - 0.25 E - 1 JE - 0.25 Total - 1.5 |
$12K (training, conferences) |
| Involvement in corporate initiatives and representing the Evaluation Unit within CIHR and externally (e.g. support to SPM, EMC, AC; preparation of briefing notes, status of evaluations) | EM - 0.25 E - 1 Total - 1.25 |
$0 |
| Total | 9 mths | $12K |
2.3 CIHR 5-Year Evaluation Schedule (2010/11 - 2014/15)
Green cells relate to the planning of evaluations, the blue to the conduct of the study at the end of which an evaluation report is produced (presented as X). The PAA number is given in brackets.
5-Year Evaluation Schedule (2010/11 - 2014/15)
2010-2011
Planning an Evaluation
CIHR Programs
1.1.2 Open Operating Grant Program (Q4)
1.4.3.1 Strategic Initiatives Programs Including Regenerative Medicine & Nanomedicine Initiative & Canadian Light Source Program (Q1)
1.4.4 Partnerships for Health System Improvement ((PHSI & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Q3)
Tri-Agency Programs and Horizontal Initiatives
1.3.2.3 Centres of Excellence for Commercialization and Research Program (Q3)
1.4.1.5 National Anti-Drug Strategy (NADS) Treatment Research Initiative (Q3)
Ongoing Evaluation
CIHR Programs
1.1.5 Open Operating Grant Programs (Q1, Q2)
1.1.6 Randomized Controlled Trials Program (Q1, Q2, Q3)
1.2.1.1 Salary Support Programs (Q1, Q2, Q3)
1.4.4.1 Strategic Initiatives Programs Including Regenerative Medicine & Nanomedicine Initiative & Canadian Light Source Program (Q2, Q3, Q4)
1.4.5 Partnerships for Health System Improvement ((PHSI & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Q4)
Tri-Agency Programs and Horizontal Initiatives
2.1.3 Canada Research Chairs Program (Q1, Q2, Q3)
1.3.2.2 Business-Led Centres of Excellence (Q3, Q4)
1.3.2.3 Centres of Excellence for Commercialization and Research Program (Q4)
1.4.1.5 National Anti-Drug Strategy (NADS) Treatment Research Initiative (Q4)
Evaluation Report
CIHR Programs
1.1.7 Open Operating Grant Programs (Q3)
1.1.8 Randomized Controlled Trials Program (Q4)
1.2.1.1 Salary Support Programs (Q4)
Tri-Agency Programs and Horizontal Initiatives
2.1.4 Canada Research Chairs Program (Q4)
2011-2012
Planning an Evaluation
CIHR Programs
1.3.1 Collaborative Health Research Partnerships Program (CHRP) & Proof of Principle Program (POP I and POP II) (Q1)
Tri-Agency Programs and Horizontal Initiatives
1.2.1.3 Canada Excellence Research Chairs Program (Q3, Q4)
1.2.2.2 Canada Graduate Scholarships Program (Q1, Q2)
1.2.2.3 Vanier Canada Graduate Scholarships Program (Q1, Q2)
1.4.1.3 HIV/AIDS Research Initiative (Q1)
1.4.1.7 Drug Safety and Effectiveness Network (Q4)
Ongoing Evaluations
CIHR Programs
1.1.1 Open Operating Grants Program (Q1, Q2, Q3)
1.3.1 Collaborative Health Research Partnerships Program (CHRP) & Proof of Principle Program (POP I and POP II) (Q2, Q3, Q4)
1.4.1.1 Strategic Initiatives Programs Including Regenerative Medicine & Nanomedicine Initiative & Canadian Light Source Program (Q1, Q2)
1.4.2 Partnerships for Health System Improvement ((PHSI & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Q1, Q2)
Tri-Agency Programs and Horizontal Initiatives
1.2.2.2 Canada Graduate Scholarships Program (Q3, Q4)
1.2.2.3 Vanier Canada Graduate Scholarships Program (Q3, Q4)
1.3.2.3 Centres of Excellence for Commercialization and Research Program (Q1, Q2)
1.4.1.3 HIV/AIDS Research Initiative (Q2, Q3, Q4)
1.4.1.5 National Anti-Drug Strategy (NADS) Treatment Research Initiative (Q1, Q2)
Evaluation Report
CIHR Programs
1.1.1 Open Operating Grants Program (Q4)
1.4.1.1 Strategic Initiatives Programs Including Regenerative Medicine & Nanomedicine Initiative & Canadian Light Source Program (Q3)
1.4.2 Partnerships for Health System Improvement ((PHSI & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Q3)
Tri-Agency Programs and Horizontal Initiatives
1.3.2.2 Business-Led Centres of Excellence (Q1)
1.3.2.3 Centres of Excellence for Commercialization and Research Program (Q3)
1.4.1.5 National Anti-Drug Strategy (NADS) Treatment Research Initiative (Q3)
2012-2013
Planning an Evaluation
CIHR Programs
1.1.1 Open Operating Grants Program (Q2)
1.4.1.1 Strategic Initiatives Programs Including Regenerative Medicine & Nanomedicine Initiative & Canadian Light Source Program (Q4)
1.4.1.2 Institute Support Grants (Q2)
Tri-Agency Programs and Horizontal Initiatives
1.3.2.1 Networks of Centres of Excellence (Q3)
1.3.2.2 Business-Led Networks of Centres of Excellence (Q3)
1.4.1.4 Pandemic Preparedness Strategic Research Initiative (Q1, Q2)
Ongoing Evaluations
CIHR Programs
1.1.1 Open Operating Grants Program (Q3, Q4)
1.4.1.2 Institute Support Grants (Q3, Q4)
Tri-Agency Programs and Horizontal Initiatives
1.2.1.3 Canada Excellence Research Chairs Program (Q1, Q2, Q3, Q4)
1.2.2.2 Canada Graduate Scholarships Program (Q1, Q2)
1.2.2.3 Vanier Canada Graduate Scholarships Program (Q1, Q2)
1.4.1.4 Pandemic Preparedness Strategic Research Initiative (Q3, Q4)
1.4.1.7 Drug Safety and Effectiveness Network (Q1, Q2, Q3)
Evaluation Report
CIHR Programs
1.3.1 Collaborative Health Research Partnerships Program (CHRP) & Proof of Principle Program (POP I and POP II) (Q1)
Tri-Agency Programs and Horizontal Initiatives
1.2.2.2 Canada Graduate Scholarships Program (Q3)
1.2.2.3 Vanier Canada Graduate Scholarships Program (Q3)
1.4.1.3 HIV/AIDS Research Initiative (Q1)
1.4.1.7 Drug Safety and Effectiveness Network (Q4)
2013-2014
Planning an Evaluation
CIHR Programs
1.1.2 Randomized Controlled Trials Program (Q4)
1.2.1.1 Salary Support Programs (Q4)
Ongoing Evaluations
CIHR Programs
1.1.1 Open Operating Grants Program (Q1)
1.4.1.1 Strategic Initiatives Programs Including Regenerative Medicine & Nanomedicine Initiative & Canadian Light Source Program (Q1, Q2, Q3)
1.4.1.2 Institute Support Grants (Q1)
Tri-Agency Programs and Horizontal Initiatives
1.3.2.1 Networks of Centres of Excellence (Q1, Q2, Q3)
1.3.2.2 Business-Led Networks of Centres of Excellence (Q1, Q2, Q3)
1.4.1.4 Pandemic Preparedness Strategic Research Initiative (Q1, Q2)
Evaluation Report
CIHR Programs
1.1.1 Open Operating Grants Program (Q2)
1.4.1.1 Strategic Initiatives Programs Including Regenerative Medicine & Nanomedicine Initiative & Canadian Light Source Program (Q4)
1.4.1.2 Institute Support Grants (Q2)
Tri-Agency Programs and Horizontal Initiatives
1.2.1.3 Canada Excellence Research Chairs Program (Q1)
1.3.2.1 Networks of Centres of Excellence (Q4)
1.3.2.2 Business-Led Networks of Centres of Excellence (Q4)
1.4.1.4 Pandemic Preparedness Strategic Research Initiative (Q3)
2014-2015
Planning an Evaluation
CIHR Programs
1.1.1 Open Operating Grants Program (Q1)
1.4.1.6 Strategy on Patient-Oriented Research Initiative (Q1)
Ongoing Evaluations
CIHR Programs
1.1.1 Open Operating Grants Program (Q2, Q3)
1.1.2 Randomized Controlled Trials Program (Q1, Q2, Q3)
1.2.1.1 Salary Support Programs (Q1, Q2, Q3)
1.4.1.6 Strategy on Patient-Oriented Research Initiative (Q2, Q3)
Evaluation Report
CIHR Programs
1.1.1 Open Operating Grants Program (Q4)
1.1.2 Randomized Controlled Trials Program (Q4)
1.2.1.1 Salary Support Programs (Q4)
1.4.1.6 Strategy on Patient-Oriented Research Initiative (Q4)
Program Evaluation - Sampling within PAA Categories
A significant challenge for Evaluation at CIHR is that the agency has a large number of small programs within each PAA category. This reflects the unique structure of the organization with its 13 Institutes and wide range of programs and initiatives. However, the large number of small programs within each PAA category brings with it the risk that findings and recommendations from program evaluations will be at too high a level to provide meaningful and actionable results for managers.
In order to mitigate this issue, a sampling approach to evaluating programs within some PAA categories will be taken. Decisions on which programs to sample will be based on the following considerations and discussed with program managers:
- Materiality — the evaluation should cover the largest proportion of program spending feasible; the largest programs within each PAA category will reflect both the highest potential risk and the most significant programs for the organization;
- Strategic considerations — a program may be included in the evaluation based on its significance to informing future program redesign or importance to senior management;
- Lifecycle — for example, programs that are scheduled to sunset;
- Performance — including programs where there is an existing concern about performance.
It should be noted that this sampling strategy will only be employed with CIHR programs that do not involve other departments or agencies and which do not have terms and conditions specifying a specific evaluation approach or timing.
Table 5 - Detailed 5-Year Evaluation Plan and Resource Schedule
Note: where an evaluation spans two fiscal years, this appears as a separate entry per year on the resource schedule. Annual resource allocation allows for scope for evaluations to be added and resourced in future years as required, for example, through program redesign or the creation of new programs. Horizontal/Tri-Agency programs and initiatives are highlighted in red.
| Link to PAA | Title of Proposed Evaluation | Evaluation Approach | Schedule Rationale | Risk Level | Tri-Agency/ Horizontal Evaluation | Lead | Start Date/ Planned Start Date | Planned Approval Date | Total Planned Spending (including Gs&Cs)15 | Evaluation resources required | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| $ Millions | External Costs ($)16 | Internal Costs (Person Months)17 | |||||||||
| Fiscal Year 2010-2011 | |||||||||||
| 1.1.1 | Open Operating Grant Program | Evaluative study of historical outputs, outcomes and impacts) | Completion of existing evaluative study | High | No | CIHR | Jan/09 (Continued from previous FY) |
Oct/ 10 | $412.2 | $0 | 2.5 |
| 1.1.1 | Open Operating Grant Program | Relevance & Effectiveness | Large materiality and high risk within program suite, aspects of OGP are evaluated annually. The open suite of programs are also being redesigned; evaluation should help inform redesign | High | No | CIHR | Jan/11 | March/12 (Continued in following FY) |
$412.2 | $0 | 2.5 |
| 1.1.2 | Randomized Controlled Trials Program | Summative | Completion of existing evaluation | High | No | CIHR | Jul/09 (Continued from previous FY) |
Mar/11 | $31.4 | $0 | 2.5 |
| 1.2.1.1 | Salary Support Programs | Summative | Evaluation coverage of all programs over 5 years; Open suite of programs redesign; risk level | High | No | CIHR | Apr/10 | Mar/11 | $15.8 | $25K (Evaluation core budget) |
13.5 |
| 1.2.1.2 | Canada Research Chairs Program | Summative | Program Terms and Conditions | n/a | Yes | SSHRC | Feb/09 Continued from previous FY) |
Mar/11 | $106.9 | From secretariat ($288K) | 6 |
| 1.3.2.2 | Business-Led Networks of Centres of Excellence | Relevance & Effectiveness | Program Terms and Conditions | n/a | Yes | NSERC | Oct/10 | Jun/11 (Continued in following FY) |
$1.8 | $0 | 2 |
| 1.3.2.3 | Centres of Excellence for Commercialization and Research Program | Summative | Results-based Management and Accountability Framework (RMAF) | n/a | Yes | NSERC | Oct/10 | Dec/11 (Continued in following FY) |
$4.5 | From secretariat ($200K) | 1 |
| 1.4.1.1 | Strategic Initiatives Programs (including Regenerative Medicine and Nanomedicine Initiative & Canadian Light Source Program) |
Summative | Evaluation coverage via review of 13 CIHR Institutes being conducted through 2011 CIHR International Review; Timing of evaluation to coincide with IR. Informing program reform. | High | No | CIHR | Apr/10 | Dec/11 (Continued in following FY) |
$175.6 | $0 | 12 |
| 1.4.1.5 | National Anti-Drug Strategy (NADS) Treatment Research Initiative | Summative (Impact) | Results-based Management Accountability Framework (RMAF) | Low | Yes | Justice Canada | Oct/10 | Dec/11 (Continued in following FY) |
$0.9 | $200k (not from Evaluation core budget) | 1 |
| 1.4.2 | Partnerships for Health System Improvement (PHSI) & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Knowledge Translation Programs) |
Summative | Evaluation coverage of largest active knowledge translation initiatives and programs | Medium | No | CIHR | Aug/10 | Dec/11 (Continued in following FY) |
$16.4 | $25K (Evaluation core budget) |
6 |
| n/a | Other ongoing evaluation unit activities (e.g. consultation , implementation of TBS policy, involvement in corporate initiatives etc) | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | 11 |
| Total Fiscal Year | $50K | 60 | |||||||||
| Link to PAA | Title of Proposed Evaluation | Evaluation Approach | Schedule Rationale | Risk Level | Tri-Agency/ Horizontal Evaluation | Lead | Start Date/ Planned Start Date | Planned Approval Date | Total Planned Spending (including Gs&Cs)18 | Evaluation resources required | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| $ Millions | External Costs ($)19 | Internal Costs (Person Months)20 | |||||||||
| Fiscal Year 2011-2012 | |||||||||||
| 1.1.1 | Open Operating Grant Program | Relevance & Effectiveness | Large materiality and high risk within program suite, aspects of OGP are evaluated annually. The open suite of programs are also being redesigned; evaluation should help inform redesign | High | No | CIHR | Jan/11 (Continued from previous FY) |
Mar/12 | $412.2 | $25K (Evaluation core budget) |
13 |
| 1.2.1.3 | Canada Excellence Research Chairs Program | Summative | Data availability and sufficient elapsed time to assess outcomes | n/a | Yes | SSHRC | Nov/11 | Apr/13 (Continued in following FY) |
$TBD | From secretariat ($150K) | 4 |
| 1.2.2.2 | Canada Graduate Scholarships Program | Summative | Availability of data; alignment with Vanier CGS evaluation; availability of evaluation budget | n/a | Yes | CIHR | Apr/11 | Nov/12 (Continued in following FY) |
$37.0 | $70-80K from Evaluation core budget ($210K - $240K in total for Tri-Agencies) | 7 |
| 1.2.2.3 | Vanier Canada Graduate Scholarships Program | Summative | Availability of data; alignment with CGS evaluation; availability of evaluation budget | n/a | Yes | CIHR | $2.9 | ||||
| 1.3.1 | Collaborative Health Research Partnerships Program (CHRP) & Proof of Principle Program (POP I and II) (Research Commercialization Programs) |
Summative | Availability of data; evaluation coverage of largest active programs and initiatives | Medium | No | CIHR | Apr/11 | Jun/12 (Continued in following FY) |
$18.8 | $25K (Evaluation core budget) |
11 |
| 1.3.2.2 | Business-Led Networks of Centres of Excellence | Relevance & Effectiveness | Program Terms and Conditions | n/a | Yes | NSERC | Oct/10 (Continued from previous FY) |
Jun/11 | $1.8 | From Secretariat ($TBD) | 3 |
| 1.3.2.3 | Centres of Excellence for Commercialization and Research Program | Summative | Results-based Management and Accountability Framework (RMAF) | n/a | Yes | NSERC | Oct/10 (Continued from previous FY) |
Dec/11 | $4.5 | $0 | 4 |
| 1.4.1.1 | Strategic Initiatives Programs (Including Regenerative Medicine and Nanomedicine Initiative & Canadian Light Source Program) |
Summative | Evaluation coverage via review of 13 CIHR Institutes being conducted through 2011 CIHR International Review; Timing of evaluation to coincide with IR. Informing program reform. | High | No | CIHR | Apr/10 (Continued from previous FY) |
Dec/11 | $175.6 | $0 | 8 |
| 1.4.1.3 | HIV/AIDS Research Initiative | Summative | Results-based Management Accountability Framework (RMAF) | n/a | Yes | Public Health Agency of Canada | Apr/11 | Jun/12 (Continued in following FY) |
$22.8 | $25K (Evaluation core budget) |
8 |
| 1.4.1.7 | Drug Safety and Effectiveness Network | Review of relevance and effectiveness | Terms and conditions; performance measurement and evaluation framework | n/a | Yes | CIHR/ Health Canada | Jan/12 | Mar/13 (Continued in following FY) |
$5.4 (CIHR only, annual average of five years) | From secretariat ($150-200K) |
4 |
| 1.4.1.5 | National Anti-Drug Strategy (NADS) Treatment Research Initiative | Summative (Impact) |
Results-based Management Accountability Framework (RMAF) | Low | Yes | Justice Canada | Oct/10 (Continued from previous FY) |
Dec/11 | $0.9 | $200k (not from Evaluation core budget) | 1.5 |
| 1.4.2 | Partnerships for Health System Improvement (PHSI) & Meetings, Planning and Dissemination Grants Program & Knowledge to Action Grant Program (Knowledge Translation Programs) |
Summative | Evaluation coverage of largest active knowledge translation initiatives and programs | Medium | No | CIHR | Aug/10 (Continued from previous FY) |
Dec/11 | $16.4 | $0 | 7.5 |
| n/a | Other ongoing evaluation unit activities (e.g. consultation , implementation of TBS policy, involvement in corporate initiatives etc) | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | 18 |
| Total Fiscal Year 2011-2012 | $145K-$155K | 89 | |||||||||
| Link to PAA | Title of Proposed Evaluation | Evaluation Approach | Schedule Rationale | Risk Level | Tri-Agency/ Horizontal Evaluation | Lead | Start Date/ Planned Start Date | Planned Approval Date | Total Planned Spending (including Gs&Cs)21 | Evaluation resources required | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| $ Millions | External Costs ($)22 | Internal Costs (Person Months)23 | |||||||||
| Fiscal Year 2012-2013 | |||||||||||
| 1.1.1 | Open Operating Grant Program | Evaluative study | Large materiality and high risk within program suite, aspects of OGP are evaluated annually. | High | No | CIHR | Jul/12 | Sep/13 (Continued in following FY) |
$412.2 | $25K (Evaluation core budget) |
8 |
| 1.2.1.3 | Canada Excellence Research Chairs Program | Summative | Data availability and sufficient elapsed time to assess outcomes | n/a | Yes | SSHRC | Nov/11 (Continued from previous FY) |
Apr/13 (Continued in following FY) |
$TBD | From secretariat ($150K) |
3 |
| 1.2.2.2 | Canada Graduate Scholarships Program | Summative | Availability of data; alignment with Vanier CGS evaluation; availability of evaluation budget |
n/a | Yes | CIHR | Apr/11 (Continued from previous FY) |
Nov/12 | $37.0 | $0 | 6 |
| 1.2.2.3 | Vanier Canada Graduate Scholarships Program | Summative Availability of data; alignment with CGS evaluation; availability of evaluation budget |
n/a | Yes | CIHR | $2.9 | |||||
| 1.3.1 | Collaborative Health Research Partnerships Program (CHRP) & Proof of Principle Program (POP I and II) (Research Commercialization Programs) |
Summative | Availability of data; evaluation coverage of largest active programs and initiatives | Medium | No | CIHR | Apr/11
(Continued from previous FY) |
Jun/12 | $18.8 | $0 | 4 |
| 1.3.2.1 | Networks of Centres of Excellence Program | Summative | Evaluation coverage and timing of previous evaluations; data availability; longevity of program; timing of BL-NCE evaluation |
n/a | Yes | NSERC | Oct/12 | Jan/14 (Continued in following FY) |
$28.4 | From secretariat ($250K) |
2 |
| 1.3.2.2 | Business-Led Networks of Centres of Excellence | Summative | Data availability; timing of NCE evaluation |
n/a | Yes | NSERC | $1.8 | ||||
| 1.4.1.1 | Strategic Initiatives Program | Review of relevance and effectiveness | Timing with regards to the implementation of a new approach to managing CIHR Strategic Funding; risk and high materiality | High | No | CIHR | Jan/13 | Mar/14 (Continued in following FY) |
$175.6 | $25k | 7 |
| 1.4.1.2 | Institute Support Grants | Summative | Low risk level and materiality; Timing with 2011 CIHR International Review |
Low | No | CIHR | Jul/12 | Sep/13 (Continued in following FY) |
$13.4 | $0 | 6 |
| 1.4.1.3 | HIV/AIDS Research Initiative | Summative | Results-based Management Accountability Framework (RMAF) | n/a | Yes | Public Health Agency of Canada | Apr/11 (Continued from previous FY) |
Jun/12 | $22.8 | $0 | 4 |
| 1.4.1.4 | Pandemic Preparedness Strategic Research Initiative | Summative | Program sun-setting in 2011 | n/a | Yes | Public Health Agency of Canada | Apr/12 | Oct/13 (Continued in following FY) |
$10.6 | $0 | 6 |
| 1.4.1.7 | Drug Safety and Effectiveness Network | Review of relevance and effectiveness | Terms and conditions; performance measurement and evaluation framework |
n/a | Yes | CIHR/ Health Canada | Jan/12 (Continued from previous FY) |
Mar/13 | $5.4 (CIHR only, annual average of five years) | From secretariat ($150-200K) | 5 |
| n/a | Other ongoing evaluation unit activities (e.g. consultation , implementation of TBS policy, involvement in corporate initiatives etc) | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | 18 |
| Total Fiscal Year 2012-2013 | $50K | 69 | |||||||||
| Link to PAA | Title of Proposed Evaluation | Evaluation Approach | Schedule Rationale | Risk Level | Tri-Agency/ Horizontal Evaluation | Lead | Start Date/ Planned Start Date | Planned Approval Date | Total Planned Spending (including Gs&Cs)24 | Evaluation resources required | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| $ Millions | External Costs ($)25 | Internal Costs (Person Months)26 | |||||||||
| Fiscal Year 2013-2014 | |||||||||||
| 1.1.1 | Open Operating Grant Program | Evaluative study | Large materiality and high risk within program suite, aspects of OGP are evaluated annually. | High | No | CIHR | Jul/12 (Continued from previous FY) |
Sep/13 | $412.2 | $25K (Evaluation core budget) |
6.5 |
| 1.1.2 | Randomized Controlled Trials Program | Summative | Evaluation coverage of all programs over 5 years; risk level | High | No | CIHR | Jan/14 | Mar/15 (continued in following FY) |
$31.4 | $0 | 3 |
| 1.2.1.1 | Salary Support Programs | Summative | Evaluation coverage of all programs over 5 years; risk level | High | No | CIHR | Jan/14 | Mar/15 (Continued in following FY) |
$15.8 | $0 | 3 |
| 1.3.2.1 | Networks of Centres of Excellence Program | Summative | Evaluation coverage and timing of previous evaluations; data availability; longevity of program; timing of BL-NCE evaluation | n/a | Yes | NSERC | Oct/12 (Continued from previous FY) |
Jan/14 | $28.4 | From secretariat ($250K) |
5 |
| 1.3.2.2 | Business-Led Networks of Centres of Excellence | Summative | Data availability; timing of NCE evaluation | n/a | Yes | NSERC | $1.8 | ||||
| 1.4.1.1 | Strategic Initiatives Program | Review of relevance and effectiveness | Timing with regards to the implementation of a new approach to managing CIHR Strategic Funding; risk and high materiality | High | No | CIHR | Jan/13 (Continued from previous FY) |
Mar/14 | $175.6 | $25K (Evaluation core budget) |
9.5 |
| 1.4.1.2 | Institute Support Grants | Summative | Low risk level and materiality; Timing with 2011 CIHR International Review | Low | No | CIHR | Jul/12 (Continued from previous FY) |
Sep/13 | $13.4 | $25K (Evaluation core budget) |
5 |
| 1.4.1.4 | Pandemic Preparedness Strategic Research Initiative | Summative | Program sun-setting in 2011 | n/a | Yes | Public Health Agency of Canada | Apr/12 (Continued from previous FY) |
Oct/13 | $10.6 | $25K (Evaluation core budget) |
7.5 |
| n/a | Other ongoing evaluation unit activities (e.g. consultation , implementation of TBS policy, involvement in corporate initiatives etc) | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | 18 |
| Total Fiscal Year 2013-2014 | $100K | 57.5 | |||||||||
| Link to PAA | Title of Proposed Evaluation | Evaluation Approach | Schedule Rationale | Risk Level | Tri-Agency/ Horizontal Evaluation | Lead | Start Date/ Planned Start Date | Planned Approval Date | Total Planned Spending (including Gs&Cs)27 | Evaluation resources required | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| $ Millions | External Costs ($)28 | Internal Costs (Person Months)29 | |||||||||
| Fiscal Year 2014-2015 | |||||||||||
| 1.1.1 | Open Operating Grant Program | Summative | Large materiality and high risk within program suite, aspects of OGP are evaluated annually. | High | No | CIHR | Apr/14 | Mar/15 | 412.2 | $25k | 13 |
| 1.1.2 | Randomized Controlled Trials Program | Summative | Evaluation coverage of all programs over 5 years; risk level | High | No | CIHR | Jan/14 (continued from previous FY) |
Mar/15 | $31.4 | $25K | 13 |
| 1.2.1.1 | Salary Support Programs | Summative | Evaluation coverage of all programs over 5 years; risk level | High | No | CIHR | Jan/14 (continued from previous FY) |
Mar/15 | $15.8 | $25K | 13 |
| 1.4.1.6 | Strategy on Patient-Oriented Research (SPOR) Initiative | Summative | Program lifecycle (started in 2009??); availability of data | Medium | No | CIHR | Apr/14 | Mar/15 | $7.6 | $25K | 13 |
| n/a | Other ongoing evaluation unit activities (e.g. consultation , implementation of TBS policy, involvement in corporate initiatives etc) | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | 21 |
| Total Fiscal Year 2014-2015 | $100K | 73 | |||||||||
3. Risks and Success Factors
The proposed Evaluation Plan consumes the entire capacity of the Evaluation function in 2010/11. The successful (i.e., complete and timely) implementation of the Plan is predicated on effective collaboration with evaluation clients, appropriate resources, effective evaluation methodologies and structured project management. The following tables depict success factors and success strategies associated with the implementation of the EP and an assessment of potential risks to implementation and mitigation strategies to address these.
3.1 Success Factors
Table 6 - Success Factors
| Success Factors | Success Strategies |
|---|---|
| 1. Effective collaboration with evaluation clients |
|
| 2. Appropriate resources |
|
| 3. Effective evaluation methodologies |
|
| 4. Effective project management |
|
3.2 Risk Assessment
Table 7 - Risk Assessment
| Risk | Impact | Probability | Mitigation |
|---|---|---|---|
| Changes to the CIHR Performance Activity Architecture in 2010/11.
This may have impacts on assessments of overall program coverage as evaluations are structured according to PAA category. |
Medium | High | The Evaluation Unit will work closely with CIHR colleagues as the PAA is being redesigned, assessing coverage and making changes to planned evaluations as required. Recording these changes and describing their rationale will enable coverage to be clearly demonstrated to TBS in future evaluation plans. |
| Delay in completion of elements of the International Review could impact on timely completion of the Strategic Initiatives Evaluation that is scheduled to report in December 2011 | Medium | Low | Full involvement in International Review processes to assess risks on an ongoing basis. Develop alternative data collection strategies. |
| Insufficient FTE or financial resources to conduct planned evaluations | High | Low | Required contracting budgets to be planned and requested in advance to allow for business case to be made. Staffing to be completed by Fall 2010 and retention strategies devised. Resource requirements and usage clearly communicated to management and EC. |
| Program scheduled for evaluation cannot be evaluated - e.g. due to a lack of appropriate performance measurement data, programs with widely varying objectives falling into same PAA etc. | High | Medium | Conduct preliminary evaluability assessments well before the evaluation is due to be conducted. Work with program managers on an ongoing basis to ensure that appropriate data is being collected. |
- Modified: