Mid-Term Evaluation of the Institute of Human Development, Child and Youth Health (IHDCYH)
Executive Summary
December 2005
Table of Contents
1. Executive Summary
1.1 Overview of CIHR
1.2 Overview of the Institute of Human Development, Child and Youth Health (IHDCYH)
1.2.1 Mandate
1.2.2 Strategic Research Priorities
1.3 Evaluation Objectives and Issues
1.3.1 Methodology
2. Evaluation Results
2.1 Relevance
2.2 Effectiveness
2.3 Delivery
3. Recommendations
4. Management Response
1. Executive Summary
1.1 Overview of CIHR
The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
CIHR was created under The CIHR Act that came into force on June 7, 2000.
Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).
In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:
- outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
- outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
- transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.
These strategic outcomes will be enabled through:
- effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
- organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.
CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.
CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and social, cultural, environmental and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.
Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.
CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).
1.2 Overview of the Institute of Human Development, Child and Youth Health (IHDCYH)
IHDCYH's mission is to "promote and facilitate research at the highest international standards in reproductive and developmental biology, pregnancy and birth, and the health and development of newborns, children, youth and their families."
1.2.2 Strategic Research Priorities
IHDCYH selected the following six strategic research priorities on which to focus its strategic initiatives:
- pre- and post-implantation health;
- reproductive aging;
- fetal growth and preterm birth;
- congenital abnormalities;
- developmental trajectories of children and youth; and
- causes, prevention and treatment of acute and chronic illness in children and youth.
1.3 Evaluation Objectives and Issues
The Common Performance Measurement and Evaluation Framework (henceforth referred to as the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:
- to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
- to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.
The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:
- Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
- Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
- Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
- Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?
The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.
1.3.1 MethodologyThe evaluation consisted of four main lines of evidence:
- a review of documents and administrative data relating to IHDCYH;
- 40 key informant interviews with IHDCYH staff and IAB members, researchers and students, and stakeholders and partners;
- case studies of two IHDCYH initiatives; and
- a telephone survey of 127 funded and 39 non-funded researchers affiliated with IHDCYH.
2. Evaluation Results
2.1 Relevance
Evidence gathered as part of this evaluation leads to the conclusion that IHDCYH's mandate and strategic research priorities are generally appropriate. IHDCYH is considered to be an appropriate and needed mechanism to develop Canadian capacity, research excellence, and knowledge translation in its research domains, although its effectiveness is seen as being tied to its ability to partner with other Institutes and organisations. No alternative is offered to IHDCYH. Its role is considered particularly important to ensure that sufficient focus is put on children, youth, reproductive and maternal health research.
IHDCYH is also seen as functioning well as a virtual Institute, in spite of management and communication challenges. It did, however, experience a temporary loss of momentum as a result of its SD and staff transition, a risk inherent to the current CIHR virtual model.
2.2 Effectiveness
To date IHDCYH has made significant progress in the areas of capacity building, contribution to CIHR's transformative vision, and contribution to CIHR's ethics mandate. It has managed an SD and staff transition and has addressed the creation of effective partnerships.
Although a number of knowledge translation activities were funded by IHDCYH, there is general agreement among key informants that the Institute has made a limited contribution to knowledge translation.
To date IHDCYH has addressed only four of its six strategic research priorities. Its Strategic Plan has not been implemented entirely as intended due to factors recognized by most staff, IAB members, researchers and stakeholders as being outside of the Institute's control.
Given the early stage of the development of IHDCYH and the nature of health research, it is difficult to measure the overall impact of the Institute's efforts at this time, particularly in light of the absence of data collected on the impacts of the Institute's research and training grants. Although IHDCYH has led or contributed to a number of promising strategic research initiatives, there is consensus among key informants that it is too early to measure the impact of these on new knowledge creation. There is some evidence that appropriate activities are being funded that address areas of priority for IHDCYH's community of researchers and stakeholders, and that these activities are likely to lead to positive impacts in the areas of reproductive biology, pregnancy and birth, and the health and development of newborns, children, youth and their families.
2.3 Delivery
Overall, evidence shows that appropriate mechanisms were put in place to ensure the implementation of the Institute's activities: effective planning mechanisms were implemented at the outset by the Institute and before each large RFA it issued; IHDCYH is currently conducting consultations to prepare a new strategic plan; and its IAB appears to be generally functioning well.
IHDCYH's consultation activities and communication efforts have not been as elaborate as they could be due to budget and time limitations. Evidence indicates that the Institute's communication efforts were not entirely successful in involving its community and that it generally did not achieve very high visibility. IHDCYH's lack of visibility has hindered its capacity to effectively communicate with its target clientele. This limitation is attributable in part to the transition experienced by the Institute. The Institute suffered from limited support from what it perceived as limited support from CIHR central during the transition.
Although the balance between funding for investigator-initiated and strategic research (70:30) is deemed adequate by most key informants, overall CIHR funding for IHDCYH-related research and for the Institute's strategic initiatives is viewed as insufficient in light of the Institute's strategic priorities and the need to sustain new capacity built.
3. Recommendations
Following are recommendations that emerge from this mid-term evaluation of IHDCYH. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.
Recommendation 1:
The Institute has been found to be doing well and is encouraged to continue the following:
- sustain its efforts in the areas of research excellence, capacity development and funding strategic priorities;
- maintain the current structure and operation of the IAB;
- maintain current planning mechanisms, as they are effective;
- sustain its efforts in contributing to CIHR's ethics mandate; and
- sustain its efforts in contributing to the transformative vision of CIHR through its IAB representation, its RFA requirements, and its support to strategic initiatives favouring the integration of researchers from non-biomedical research areas.
The following areas are ones in which it is recommended the Institute take action to improve:
Recommendation 2:
Consultation and Communication - It is recommended that IHDCYH develop and implement an overall communications strategy to ensure its stakeholder communities are aware of the results of its activities.
Recommendation 3:
Collaboration and Partnerships - IHDCYH is encouraged to continue its increased efforts in creating partnerships and leveraging of resources with other Institutes and stakeholder organizations and to encourage other Institutes to address research issues of particular relevance to children and youth.
Recommendation 4:
Knowledge Translation - IHDCYH should review its efforts in support of KT. The efforts expended by IHDCYH in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that IHDCYH are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.
Recommendation 5:
Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.
4. Management Response
Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:
| Recommendation | Response | Action Plan |
| 1. Consultation and Communication - It is recommended that IHDCYH develop and implement an overall communications strategy to ensure its stakeholder communities are aware of the results of its activities. | Agree | IHDCYH will prepare, implement, and evaluate a communications plan for its stakeholders and the general public. For example IHDCYH will:
|
| 2. Knowledge Translation - IHDCYH should review its efforts in support of KT. The efforts expended by IHDCYH in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that IHDCYH are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT. | Agree |
|
| 3. Collaboration and Partnerships - IHDCYH is encouraged to continue its increased efforts in creating partnerships and leveraging of resources with other Institutes and stakeholder organizations and to encourage other Institutes to address research issues of particular relevance to children and youth. | Agree | With our increased efforts to seek new partners and sustain existing partnerships, IHDCYH's personnel and financial resources are stretched to the maximum: we will attempt to maintain our current level of activities but be unable to consider further increases. IHDCYH will continue to develop an effective partnership strategy in Canada and abroad and will also manage and nurture established partnerships. For example, IHDCYH will:
|
| 4. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. | Agree | IHDCYH will establish a mechanism to monitor and evaluate progress in achieving its strategic goals and key objectives. For example, IHDCYH will Monitor RFAs more efficiently: use of funds, achievement of milestones, attainment of goals, etc. |
Supplemental content (right column)
- Modified: