Mid-Term Evaluation of the Institute of Aging (IA)
Executive Summary
December 2005
Table of Contents
1. Executive Summary
1.1 Overview of CIHR
1.2 Overview of the Institute of Aging (IA)
1.2.1 Mandate
1.2.2 Strategic Orientations and Objectives
1.2.3 Research Priorities
1.3 Evaluation Objectives and Issues
1.3.1 Methodology
2. Evaluation Results
2.1 Relevance
2.2 Effectiveness
2.3 Delivery
3. Recommendations
4. Management Response
1. Executive Summary
1.1 Overview of CIHR
The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
CIHR was created under The CIHR Act that came into force on June 7, 2000.
Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).
In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:
- outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
- outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
- transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.
These strategic outcomes will be enabled through:
- effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
- organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.
CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.
CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.
Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.
CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).
1.2 Overview of the Institute of Aging (IA)
The fundamental goal of the Institute of Aging (IA) is the advancement of knowledge in the field of aging to improve the quality of life and the health of older Canadians. The Institute supports research to promote healthy and successful aging and to address the causes, prevention, screening, diagnosis, treatment, support systems, and palliation for a wide range of conditions associated with aging. Research supported by the IA pertains to topics such as:
- Long-term health promotion policies and strategies (individual, community and population levels);
- Identification of health advantage and health risk factors related to the interaction of environments over a lifetime (cultural, social psychological, behavioural, physical, genetic);
- Demography and epidemiology: identification of determinants of health status of a population later in life and potential impacts on health services for forward planning;
- Factors which enable the elderly to remain independent: e.g., community characteristics, housing, transportation, and volunteers;
- Biological mechanisms underlying the aging process itself and conditions associated with aging;
- Geriatrics research: best practice research on care of the elderly (e.g., service delivery, co-morbidity, medication use and misuse, iatrogenic problems, rehabilitation, palliation); and
- Aging-related disorders and disability (e.g., neurodegenerative disorders, cognitive decline, reduced mobility).
1.2.2 Strategic Orientations and Objectives
The IA has five strategic orientations (each with its own specific objectives):
- Leadership in setting strategic research directions in the field of aging in Canada;
- Development and support of capacity-building research initiatives and programs in the field of aging;
- Development and support of strategic research initiatives, programs and projects on aging;
- Promotion of the importance of research on aging and of the needs of the research community in aging; and
- Facilitating the dissemination, transfer and translation of knowledge from research findings into potential applications through policies, interventions, services and products.
On the basis of consultations with its IAB, researchers and various stakeholders, the IA has identified five research priorities in the field of aging:
- Healthy and successful aging;
- Biological mechanisms of aging;
- Aging and maintenance of functional autonomy;
- Cognitive impairment in aging; and
- Health services and policy relating to older people.
1.3 Evaluation Objectives and Issues
The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:
- to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
- to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.
The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:
- Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
- Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
- Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
- Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?
The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.
The evaluation consisted of four main lines of evidence:
- a review of documents and administrative data relating to the IA;
- 44 key informant interviews with IA staff and IAB members, researchers and students, stakeholders and partners;
- case studies of three IA initiatives; and
- a telephone survey of 72 funded and 36 non-funded researchers affiliated with the IA.1
2. Evaluation Results
2.1 Relevance
The evaluation evidence indicates that the mandate and priorities of the IA are appropriate and relevant. A majority of key informants in all respondent groups regard the Institute's mandate, strategic orientations and research priorities as appropriate, though a minority feels the scope is too broad (e.g., in light of limited resources). In addition, in the survey four-fifths of funded IA researchers and almost two-thirds of non-funded researchers see a close fit of their research interests with the Institute's mandate.
IA is generally viewed as an appropriate mechanism to make a difference in its research domain of aging. Furthermore, researchers surveyed believe there is a great need for the IA to support research excellence, capacity development and funding of strategic priorities. No alternatives to the IA mechanism were identified, although some improvements were suggested (e.g., increased dialogue and partnerships to facilitate knowledge translation). The CIHR model of virtual Institutes is generally regarded as appropriate, with both advantages and disadvantages. For example, it is advantageous that Institutes are located across the country and that the model allows Scientific Directors to remain at their home institution, but on the other hand Institutes may feel disconnected from CIHR central and the transition of SDs and staff can be disruptive and result in a loss of corporate memory for an Institute.
2.2 Effectiveness
Overall, IA has performed well and has made good progress in contributing to CIHR's overall objective and in fulfilling the Institute's mandate, strategic orientations and research priorities.
Through RFAs2 (Requests for Applications), the funding of research and major initiatives such as the CLSA, NET grants and CIA Partnership, IA is making progress in creating new knowledge in the field of aging. Most funded IA researchers believe that the Institute has contributed in advancing health research to respond to opportunities/priorities and create health knowledge. Most key informants also agree that progress is being made toward knowledge creation, however, a minority feels that it is premature to conclude how much new knowledge will truly be created as a direct result of IA's efforts/funding.
IA has funded or implemented a number of KT-related activities, including a presentation to the Romanow Commission, conducting Regional Seniors' Workshops on Research, supporting the Canadian Journal on Aging and building a KT component into RFAs. Most funded IA researchers surveyed believe that the Institute's activities support KT to at least some extent and researchers are highly involved in disseminating their research findings, in particular, to other researchers and health care professionals. Key informants generally agree that IA has implemented numerous activities related to KT (e.g., involving the dissemination of knowledge to policy makers and service providers). A minority of stakeholders note, however, that the true uptake of knowledge to result in improved policies or healthcare practices for older Canadians has not yet occurred. Perceived challenges for KT include a lack of direction from CIHR central and a need for more innovative approaches to result in true knowledge uptake.
IA has contributed significantly toward capacity building through a range of initiatives to attract and retain researchers in aging and through investments in NETs, STIHRs and Training Awards, which have been increasing annually. Key informants generally agree that IA has made a strong contribution to capacity development. Similarly, the majority of funded IA researchers in the survey feel that the Institute has contributed, to at least some extent, to the development of people and the research environment. Some key informants caution, however, that it is too early and there is a lack of quantitative data to assess conclusively the IA's impact on capacity in the field of research on aging.
Key informants agree that IA is making a major contribution to the transformative vision of CIHR, in particular, due to the Institute's emphasis on interdisciplinary research in aging and key initiatives such as the CLSA, RSWRs and CIA Partnership. Moreover, in the survey, funded IA researchers report that an average of 68% of their projects are interdisciplinary. Similarly, CIHR administrative data indicate that funding for research related to the IA mandate spans all four Themes, suggesting an excellent contribution to CIHR's transformative vision.
The Institute has also made a contribution to the CIHR ethics mandate, for example, by contributing financially to joint strategic initiatives with some relation to ethics issues, by addressing ethical issues around informed consent as part of the CLSA, and by having an ethics specialist and ELSI representative on the IAB. There is, however, low awareness of the ethics mandate and the Institute's contribution in this regard among many key informants.
Key informants and researchers surveyed generally believe, and documentation confirms, that IA is progressing well relative to its mandate and priorities, in particular, by supporting the advancement of knowledge in these areas. Similarly, the majority of funded IA researchers perceive that the mandate and strategic priorities have been achieved to at least some extent. CIHR investments in research related to aging have increased steadily over the past five years, further indicating that IA is influencing research under its mandate.
Key informant interviews and the document review indicate that IA has collaborated extensively and developed numerous partnerships, linkages and alliances with a broad range of stakeholders from numerous sectors, including provincial and federal government departments/agencies, health charities, non-profit NGOs and seniors organizations, and professional associations. IA has also worked effectively with other CIHR Institutes (e.g., as partners/contributors in joint RFAs and major Institute initiatives). Perceived barriers and challenges in developing partnerships include the time/effort required to form and nurture partnerships, some partners' lack of resources/capacity, and a limited appreciation of research on the part of some partners.
IA has demonstrated leadership and has had some influence on research agendas in aging. Most key informants agree that the Institute has demonstrated leadership and influence through RFAs/funding of research in its priority areas (e.g., Alzheimer's disease, dementia) and by encouraging collaborative interdisciplinary research in aging, for instance, through New Emerging Teams. Most funded IA researchers in the survey also indicated that the Institute has influenced the research agenda in aging to at least some extent, with over half saying to a large extent. In addition, leadership and a contribution to a more coordinated research agenda are illustrated by innovative initiatives such as the CLSA and Cognitive Impairment in Aging Partnership.
2.3 Delivery
IA has used planning mechanisms effectively. The initial Strategic and Action Plan and selection of research priorities were based on extensive consultations (i.e., interviews and focus groups with a range of stakeholders from across the country) and reviewed by the IAB. Key informants generally believe that these mechanisms have been effective and that input from the community is accurately reflected in the plan and research priorities. Moreover, most researchers surveyed feel that the IA has successfully identified emerging priorities within its mandate.
The IAB has been operating effectively and contributing to the achievements of the Institute (e.g., strategic planning and key initiatives such as the Cognitive Impairment in Aging Partnership). Staff and Board members are generally satisfied with the operation, membership and effectiveness of the Board, though some minor concerns were raised about the current composition and degree of engagement of some members.
IA has consulted extensively with a wide range of stakeholders using both formal and informal methods. Key informants and researchers surveyed feel that the IA has been responsive to input and feedback and has considered this in its plans and activities. In addition, the Institute has utilized a wide range of communications approaches/vehicles, however, some key informants feel that the communications could be improved.
Factors that exert some influence on the effectiveness of IA include central CIHR functions and funding levels. Key informants expressed a range of positive and negative views on the degree of support and responsiveness of central CIHR functions; there was little consensus. Areas of concern for some interviewees include a lack of leadership on knowledge translation and ethics, operational support, and evaluation. Most key informants view the level of funding ($1M for the Institute support grant and approximately $6.5M for strategic funding) as inadequate to make a difference in all areas of IA's broad mandate, though views are variable on each type of funding. Although mixed views are expressed on the 70:30 split between funding for investigator-initiated and strategic research, many key informants regard this balance as appropriate. Still, some staff favour a higher proportion of strategic research funding whereas some researchers would like to see more emphasis on funding for investigator-initiated research.
3. Recommendations
Following are recommendations that emerge from this mid-term evaluation of IA. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.
Recommendation 1:
The Institute has been found to be doing well and is encouraged to continue the following:
- support research excellence, capacity development and funding of strategic priorities in the field of aging;
- maintain the current structure and operation of the IAB as it is functioning effectively;
- sustain efforts in consulting with researchers and a diversity of stakeholders using range of formal and informal methods;
- maintain its efforts directed toward the creation of knowledge in the field of aging, utilizing its RFAs/funding vehicles and major initiatives such the CLSA, CIA Partnership and Mobility in Aging; and
- maintain its efforts in contributing to the transformative vision of CIHR through its emphasis on interdisciplinary research covering all four Themes as well as its initiatives that support this focus as well as knowledge translation (e.g., the CLSA, RSWRs and CIA Partnership).
The following areas are ones in which it is recommended the Institute take action to improve:
Recommendation 2:
Strategic Priorities - The Institute is encouraged to continue to implement their strategic research priorities as they remain appropriate but it should also consider periodic refinements to its focus on particular priorities, based on its previous investments and successes, to ensure that all priority areas receive attention.
It is recommended that IA clarify with CIHR and other Institutes which diseases common in old age it can feasibly focus on, given its level of resources.
Recommendation 3:
Communications and Consultations - Overall, the Institute is encouraged to maintain communications activities, using a variety of approaches/vehicles. If feasible, the Institute should also consider some improvements raised by key informants in this evaluation - for example, devote more resources to communications, formulate an updated communications action plan, communicate information in understandable and practical terms to reach a broad audience including seniors beyond those that participate in workshops, and clearly communicate the Institute's progress and achievements relative to its mandate and priorities to the researcher and stakeholder community.
Recommendation 4:
Collaborations and Partnerships - While encouraged overall to maintain its work in collaborating and developing partnerships with a range of organizations and sectors, the Institute should also explore opportunities to broaden its partnerships to include, for example, more healthcare practitioner organizations, health charities, federal research funding organizations, provincial ministries and pharmaceutical companies.
Recommendation 5:
Ethics - The Institute is encouraged to further develop its efforts related to CIHR's ethics mandate, and to inform its researcher and stakeholder community about these efforts and any achievements.
Recommendation 6:
IAB Functioning - The current composition of the IAB should be examined to ensure that there is adequate representation of the NGO/practitioner community as well as both social and biomedical/bench scientists and that these members are fully engaged.
Recommendation 7:
Knowledge Translation - The IA should review its efforts in support of KT. The efforts expended by the IA in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IA are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.
Recommendation 8:
Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.
4. Management Response
Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:
| Recommendation | Response | Action Plan |
|
1. Strategic Priorities - The Institute is encouraged to continue to implement their strategic research priorities as they remain appropriate but it should also consider periodic refinements to its focus on particular priorities, based on its previous investments and successes, to ensure that all priority areas receive attention. It is recommended that IA clarify with CIHR and other Institutes which diseases common in old age it can feasibly focus on, given its level of resources. |
Agree |
Current Actions
Next Steps
Re: IA focus diseases Current Actions
Next Steps
|
| 2. Communications and Consultations - Overall, the Institute is encouraged to maintain communications activities, using a variety of approaches/vehicles. If feasible, the Institute should also consider some improvements raised by key informants in this evaluation - for example, devote more resources to communications, formulate an updated communications action plan, communicate information in understandable and practical terms to reach a broad audience including seniors beyond those that participate in workshops, and clearly communicate the Institute's progress and achievements relative to its mandate and priorities to the researcher and stakeholder community. | Agree |
Current Actions
Next Steps
|
| 3. Collaborations and Partnerships - While encouraged overall to maintain its work in collaborating and developing partnerships with a range of organizations and sectors, the Institute should also explore opportunities to broaden its partnerships to include, for example, more healthcare practitioner organizations, health charities, federal research funding organizations, provincial ministries and pharmaceutical companies. | Agree |
Current Actions
Next Steps
|
| 4. Ethics - The Institute is encouraged to further develop its efforts related to CIHR's ethics mandate, and to inform its researcher and stakeholder community about these efforts and any achievements. | Agree |
Current Actions
Next Steps
|
| 5. IAB Functioning - The current composition of the IAB should be examined to ensure that there is adequate representation of the NGO/practitioner community as well as both social and biomedical/bench scientists and that these members are fully engaged | Agree |
Current and Future Actions
|
| 6. Knowledge Translation - The IA should review its efforts in support of KT. The efforts expended by the IA in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IA are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT. | Agree |
Current Actions
Next Steps
|
| 7. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. | Agree |
Current Actions
Next Steps
|
1 Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.
2 An RFA is the mechanism for encouraging applications for funding in a priority research area and/or strategic initiative.
Supplemental content (right column)
- Modified: