Mid-Term Evaluation of the Institute of Population and Public Health (IPPH)
Executive Summary
December 2005
Table of Contents
1. Executive Summary
1.1 Overview of CIHR
1.2 Overview of the Institute of Population and Public Health (IPPH)
1.2.1 Mandate
1.2.2 Strategic Research Priorities
1.3 Evaluation Objectives and Issues
1.3.1 Methodology
2. Evaluation Results
2.1 Relevance
2.2 Effectiveness
2.3 Delivery
3. Recommendations
4. Management Response
1. Executive Summary
1.1 Overview of CIHR
The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
CIHR was created under The CIHR Act that came into force on June 7, 2000.
Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).
In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:
- outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
- outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
- transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.
These strategic outcomes will be enabled through:
- effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
- organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.
CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.
CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual, geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and social, cultural, environmental and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.
Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.
CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).
1.2 Overview of the Institute of Population and Public Health (IPPH)
The Institute of Population and Public Health (IPPH) focuses on the complex interactions that determine the health of individuals, communities and global populations. These interactions can involve biological, cultural, social and environmental factors. Support is also given to the application of that knowledge to improving the health of both populations and individuals.
1.2.2 Strategic Research Priorities
The IPPH has identified, through consultation with its community, five strategic research priorities on which its activities and funding are focused:
- Capacity Building;
- Understanding and Addressing the Impacts of Physical and Social Environments of Health;
- Characterizing and Reducing Health Disparities;
- Environmental and Genetic Determinants of Disease in Human Populations; and
- Global Health.
1.3 Evaluation Objectives and Issues
The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:
- to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
- to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.
The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:
- Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
- Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
- Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
- Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?
The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.
The evaluation consisted of four main lines of evidence:
- a review of documents and administrative data relating to the IPPH;
- 48 key informant interviews with IPPH staff and IAB members, researchers and students, stakeholders and partners;
- case studies of two IPPH initiatives; and
- a telephone survey of 97 funded and 38 non-funded researchers affiliated with the IPPH1.
2. Evaluation Results
2.1 Relevance
On the whole, the Institute's researcher and stakeholder communities are satisfied with the mandate and strategic priorities that currently guide the IPPH. While the breadth of the mandate poses challenges in terms of managing expectations across a wide array of researcher and stakeholder communities, most believe the priorities have been appropriately identified and are topical within the field. The possible exception is in the area of research on population and public health interventions (which is acknowledged by the Institute in its Strategic Plan). This evaluation and other consultations conducted by the Institute have documented the considerable interest in this area and appetite for this type of research internationally (WHO Commission on Social Determinants of Health). For many, this research represents answers to the "big questions" - what kind of policies and programs enhance the health of societies?
The diverse research domains that are encompassed by the Institute are appropriate, but again represent a challenge in terms of managing expectations. A concern is the extent to which self-designated Theme 4 researchers and grants are accurately classified. This is an issue that has been raised by the IPPH with CIHR and could explain surveyed researchers' relative lack of familiarity and reported "fit" of their research interests with the Institute's mandate.
There is a widespread belief that the IPPH is an important and appropriate mechanism to address capacity, research excellence and knowledge translation in the population and public health field and there is support for the virtual model in general. The Institute and Theme 4 research are key components of the transformative vision of CIHR. As well, from the perspective of researchers, the IPPH is one of the few bodies currently funding strategic research in population and public health - a particularly important source of research dollars considering the reported and documented (i.e., Thorngate) disproportionate access of this community to research funds through CIHR open competitions. Few suggested alternatives to the IPPH mechanism were identified.
2.2 Effectiveness
The evaluation results indicate that the IPPH has contributed to the objectives and mission of CIHR in many ways. IPPH is making progress in creating new knowledge through strategic funding through Requests for Applications (RFAs)2. The SARS II RFA and Advancing Theories, Frameworks, Methods and Measurement are notable examples. The Institute has also been a key contributor to CIHR joint strategic initiatives, with a significant portion of grants and staff support devoted to these areas. The Institute has led the Global Health Research Initiative, which has provided support to over 70 research teams.
Most key informants agree that knowledge translation (KT) has been a priority for the Institute and the IPPH has played a leadership role in this area through RFAs, a KT Casebook among others. The design of the Institute's largest initiative, the Centres for Research Development, incorporates KT in all stages of the research process, including the governance structure of the Centres. The PPH health research community has a strong tradition of knowledge translation and surveyed researchers are more apt to report that their research projects have a KT component and to disseminate their results widely. Most funded IPPH researchers also believe that the Institute's activities support KT (more so than CIHR researchers overall, referring to their own Institute). A minority of stakeholders provided suggestions as to how the Institute could improve its KT activities.
Capacity building, also a strategic priority for the Institute, has been operationalized through initiatives such as the Centres for Research Development and Summer Institutes which feature collaborations across disciplines and organizations, including inter-Institute and voluntary health organizations (VHOs). The Institute has also made significant investments through CIHR capacity development mechanisms — Interdisciplinary Capacity Enhancement (ICE) teams, New Emerging Teams (NETs), Strategic Training Initiatives in Health Research (STIHRs) and Training Awards. The Institute's contribution to the establishment of the Public Health Agency of Canada (PHAC) was also noted as important by many key informants. Most funded IPPH researchers agree that the Institute is developing people and the research environment in the field at least to some extent.
IPPH is viewed as embodying the transformative vision of CIHR - it represents and includes Theme 4 researchers, formerly outside the scope of the Medical Research Council (MRC), and researchers in this area are at the forefront in practice of interdisciplinarity and knowledge translation. Surveyed researchers reported that a significant proportion of their projects are interdisciplinary and include a KT component (higher than CIHR researchers overall). Most of the CIHR funding for research related to the IPPH's mandate is in Theme 4, and to a much less extent, Themes 1, 2 and 3.
Examples of contributions to the CIHR ethics mandate are fewer, however, the Institute has examined these issues, with others, in the context of RFAs.
Interviewees generally believe that IPPH has made progress toward achieving its mandate and priorities. The majority of surveyed researchers believe that the overall mandate of the Institute has been achieved to at least some extent and CIHR investments in population and public health research have increased significantly over the past five years (though there are vagaries in the classification of research by Theme). In terms of its strategic priorities, the Institute has made significant investments in capacity building, understanding the social and physical determinants of health and global health (though its work in global health is much less familiar to its stakeholder communities). Interestingly, global health is one of the areas where there has been significant research interest as indicated by the number of research applications received. Progress on the environmental and genetic determinants of disease has suffered with the deferral of the Birth Cohort study. This is a complex area, with few researchers in Canada and internationally, and so has presented challenges for the Institute in operationalizing this strategic priority. This is a priority area that has somewhat mixed support from the Institute's stakeholder communities.
In the area of partnerships, the Institute has been very active in building relationships with organizations within its evolving landscape - the Canadian Population Health Initiative (CPHI) at the Canadian Institute for Health Information (CIHI), the Canadian Public Health Association (CPHA) and PHAC. Most IPPH researchers indicate that the Institute has facilitated collaboration within their research community. While the Institute has formed some funding partnerships (e.g., Inter-Institute, with federal government departments in the context of the Global Health Research Initiative), this has been a challenging area for the IPPH. There are few significant research granting organizations in this field - private sector or governmental. As well, the IPPH mandate does not directly align with the mission of any of the sizeable VHOs (who associate more readily with a disease or organ-based Institute).
Key informants and surveyed researchers believe that IPPH has demonstrated leadership and is having an influence on the population and public health research agenda. The leadership of the SD was praised and the Institute's instrumental role in raising the awareness and profile of population and public health research within CIHR and in Canada was also lauded. Leadership and influence has been further illustrated through strategic funding through RFAs, fostering of collaboration and networking, and the Institute's contribution to the dramatic shifts in the broader population and public health environment (e.g., establishment of PHAC).
2.3 Delivery
There is broad approval for the planning and strategic mechanisms employed by the Institute - key informants and surveyed researchers are confident that the Institute has and continues to identify emerging priorities in the field. The initial consultation exercise - an intensive, pan-Canadian dialogue with stakeholders - generated a great deal of visibility for the Institute, as well as good will. The Institute Advisory Board (IAB) operates as an effective advisory body for the IPPH and the Institute has also employed IAB members as ambassadors to extend its communications and consultation with communities within its mandate.
Consultation is seen to be a strength of the Institute. In addition to sponsoring its own series of workshops and meetings (and conducting evaluations of these events), the Institute maintains a strong presence at scientific and professional meetings and conferences, and is an active participant in committees and working groups of its partner organizations. Communications with the Institute's various audiences is undertaken by a variety of vehicles including its newsletter, web-site, e-mail, public media, as well as informal networks and contacts.
Around planning, consultation and communications, some key informants expressed a desire for a more refined approach. Some suggestions were operational in nature (e.g., improve the Web site and newsletter). Others, however, suggested a more expansive and bold approach that would see greater promotion of Institute activities and successes, and heightened visibility of the Institute itself. This likely implies specialized communications support from outside the Institute, given its current resource limitations, as well as a clearer definition of roles vis-à-vis the communications efforts of CIHR central. Also in the area of consultation and communications, there was a desire to see the Institute continue to extend its efforts to engage certain of its constituents more fully - i.e., public health practitioners, community-based researchers, policy-makers and students.
Factors that exert some influence on the effectiveness of IPPH include central CIHR functions and funding levels. According to interviewees who were able to comment, responsiveness of central CIHR to the needs of the Institute has been uneven, with particular concerns around the handling of cross-cutters, the peer review process and guidance in the area of knowledge translation.
Funding of the IPPH and PPH research in general was a prevalent theme throughout the evaluation study. Most key informants feel that the level of funding to the Institute (currently $1M for the Institute support grant and about $6.5M for strategic funding) is inadequate in light of the IPPH's broad mandate and dual role. Great concern was also expressed about the availability of funds to Theme 4 researchers through CIHR open competitions and the appropriateness of current funding vehicles for these researchers (that have been inherited from the former MRC and were largely developed to meet the needs of the Theme 1 community). In terms of the latter, an important example cited is the lack of fit between the current funding mechanisms and the funding requirements for research on larger-scale PPH interventions (which have the ability to definitively demonstrate effectiveness of broad programs and policies as a means for improving health and reducing health care costs). Key informants clearly affirm a leadership role to IPPH in representing the interests of Theme 4 researchers to CIHR central (and other organizations such as universities) and advocating for ways to ensure that the array of funding mechanisms meet the needs of this community.
3. Recommendations
Following are recommendations that emerge from this mid-term evaluation of IPPH. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.
Recommendation 1:
The Institute has been found to be doing well and is encouraged to continue the following:
- sustain its efforts in the areas of research excellence, capacity development and funding strategic priorities, in collaboration with complementary organizations;
- maintain the current structure and operation of the IAB, as members and staff are largely satisfied with its current operation;
- maintain planning mechanisms, as they are effective;
- sustain its commitment to capacity building, using its varied programs and tools;
- sustain efforts in knowledge creation;
- maintain and expand its productive efforts in the ethics area;
- sustain its efforts in contributing to the transformative vision of CIHR through its active leadership in gaining greater recognition of population and public health, emphasis of inter-disciplinary work and knowledge translation; and
- persist in its proactive efforts vis-à-vis the review process (monitoring and addressing bias, ensuring appropriate composition and expertise of panels) and advocate for funding vehicles that are appropriate for Theme 4 research.
The following areas are ones in which it is recommended the Institute take action to improve:
Recommendation 2:
Theme 4 Responsibilities - There is a substantive degree to which IPPH has become responsible for the promotion of theme four research not only within its own community, but also across CIHR. This role, however, is not formally acknowledged. IPPH should undertake to clarify roles and responsibilities with CIHR corporate on the promotion and funding of theme four research and capacity development across CIHR.
Recommendation 3:
Strategic Priorities - With respect to strategic priorities, the Institute should consider its framing and operationalizing of current strategic priorities in the context of intervention research (while assessing trade-offs with investments in other priorities) and continue to publicize work that has been done in this area and champion an array of funding mechanisms that is appropriate to intervention research.
Recommendation 4:
Communications - While internal resources are limited for this activity, the Institute is urged to pursue a more expansive approach - with the support of CIHR central and stakeholder organizations - to clarify the ability and means for the Institute and its funded researchers to trumpet successes, document the benefits of PPH research and raise the profile of PPH research in general.
Recommendation 5:
Stakeholder Engagement - The Institute is encouraged to maintain its responsiveness to stakeholders and partners and continue to build on this foundation and extend its outreach to include other constituents, particularly those closer to delivery settings - practitioners, community-based researchers, regional public health authorities, policy-makers and students to determine their capacity, priorities, and needs.
Recommendation 6:
Collaboration and Partnerships - while encouraged overall to sustain its current course of action in partnership development, the Institute is also encouraged to explore alliances and leverage potential with a broader set of potential partners including provincial governments and regional health authorities, universities and the private sector.
Recommendation 7:
Knowledge Translation - the Institute is encouraged to sustain their priority on KT by ensuring that KT is embedded in an effective way into all Theme 4 funded research (i.e., into all aspects of the research process) and that measures are in place for dissemination/distribution of research results and KT best practices. These activities need to be pursued recognizing the role of complementary organizations such as the CPHA and the PHAC Collaborating Centres and through exchanging with practitioners about their research needs.
Recommendation 8:
Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.
4. Management Response
Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:
Summary: Overall, the Institute and its Advisory Board are very pleased with the findings of this mid-term evaluation, and look forward to implementing the recommendations, recognizing that we have to be realistic about what we can do with limited resources and what is best implemented and by whom. The results of the International Review will also provide guidance as many of the recommendations below require CIHR-wide commitment and resources to be realized as they extend well beyond the purview of IPPH.
| Recommendation | Response | Action Plan |
| 1. Theme 4 Responsibilities -There is a substantive degree to which IPPH has become responsible for the promotion of theme four research not only within its own community, but also across CIHR. This role, however, is not formally acknowledged. IPPH should undertake to clarify roles and responsibilities with CIHR corporate on the promotion and funding of theme four research and capacity development across CIHR. |
Agree |
Preamble: IPPH has had the dual responsibility of advancing a PPH strategic research and KT agenda, as well as providing advice and contributing staffing resources to support other Institute/organization efforts in theme 4, with the same level of resources. 1) Develop a discussion paper for consideration at RPPC outlining possible roles and responsibilities, feasible options with implications for IPPH, other Institutes and the organization as a whole, as well as key potential partners outside of CIHR (e.g. Public Health Agency of Canada (PHAC)). The RPPC approved paper would then be tabled at Governing Council. |
| 2. Strategic Priorities - With respect to strategic priorities, the Institute should consider its framing and operationalizing of current strategic priorities in the context of intervention research (while assessing trade-offs with investments in other priorities) and continue to publicize work that has been done in this area and champion an array of funding mechanisms that is appropriate to intervention research. |
Agree |
Preamble: Since it published its strategic plan, IPPH has identified intervention research to protect and promote the health of citizens, as a key research domain in PPH, but with limited strategic funds, and barriers to accessing matching funds from other partnered sources (e.g. organizations who would pay for the actual policy and program intervention), this objective has not been fully realized. Furthermore, many of the Institute's investments have been developmental in nature (as necessitated by modest unencumbered Institute granting budgets since 2003) but not yet resulted in longer-term intervention funding, in due part to the paucity of relevant panels in CIHR's open competition, limited research capacity, funding and available PPH funding partners in Canada. 1) Continue to advocate for and facilitate changes in CIHR's open competition to ensure equitable access by theme 4 researchers to population-level/community-level intervention research funding by:
2) Continue to support the seven Centres for Research Development in undertaking intervention research. 3) Continue to support the Canadian Public Health Association (CPHA) Conference, which provides a knowledge exchange and communication vehicle for featuring intervention research findings. |
| 3. Communications - While internal resources are limited for this activity, the Institute is urged to pursue a more expansive approach - with the support of CIHR central and stakeholder organizations - to clarify the ability and means for the Institute and its funded researchers to trumpet successes, document the benefits of PPH research and raise the profile of PPH research in general. | Agree |
PPH research in general has not been featured prominently by CIHR as a whole relative to other research domains, despite Institute efforts to do so (e.g. Day on the Hill package had no page for IPPH and IG, alone among the 13 Institutes). Greater attention by CIHR communications together with IPPH will be required to help achieve this objective. Third, relatively recent changes to the PPH landscape will provide a more conducive environment to help achieve this objective, building on current communication mechanisms (e.g. CPHA conference, KT casebook) and significant media attention already garnered by the SD. 1) IPPH will revitalize its communication plan and further identify target communication vehicles and stakeholder channels to maximize the reach and profile of IPPH research, in close consultation with CIHR Communications and partners (e.g. PHAC, CPHA) (in progress) |
| 4. Stakeholder Engagement - The Institute is encouraged to maintain its responsiveness to stakeholders and partners and continue to build on this foundation and extend its outreach to include other constituents, particularly those closer to delivery settings - practitioners, community-based researchers, regional public health authorities, policy- makers and students to determine their capacity, priorities, and needs. | Agree |
Preamble: Given the changing PPH landscape, stakeholder engagement will continue to of great importance, achieved through both formal and informal mechanisms 1) Our vision and suggested actions are already outlined in the PPH Network Action Plan (which is refreshed annually) and will continue to provide guidance to inform the implementation of our partnership and KT activities 2) IPPH will reassess with its IAB and relevant partners (Canadian Population Health Initiative, PHAC, CPHA) the need and appropriate timing to undertake another national consultation (analogous to Charting the Course (CTC)) to refresh our research and KT priorities (in light of the current resource constraints and eventual transition of the Scientific Director) |
| 5. Collaboration and Partnerships - while encouraged overall to sustain its current course of action in partnership development, the Institute is also encouraged to explore alliances and leverage potential with a broader set of potential partners including provincial governments and regional health authorities, universities and the private sector. | Agree |
Preamble: Our collaboration and partnership efforts will continue to be informed by our environmental scan of PPH partners, stakeholders and the PPH network action plan (refreshed annually), which outlines priorities for network building, partnership development in support of PPH research and its application. 1) IPPH will continue to strengthen linkages with provincial governments, public health units, regional health authorities (RHAs) and universities in areas of common interest by using indirect channels to link (e.g. CPHA, PHAC) , and by exploring possible funding mechanisms of relevance to these key organizations (e.g. possible adaptation of the Partners in Health System Improvement (PHSI) for theme 4) 2) IPPH will explore the possibility of holding a linkage and exchange event with voluntary health organizations, PPH researchers and key partners to explore ways to better engage with this community, in keeping with the CTC framework and IPPH strategic priorities |
| 6. Knowledge Translation - the Institute is encouraged to sustain their priority on KT by ensuring that KT is embedded in an effective way into all Theme 4 funded research (i.e., into all aspects of the research process) and that measures are in place for dissemination/distribution of research results and KT best practices. These activities need to be pursued recognizing the role of complementary organizations such as the CPHA and the PHAC Collaborating Centres and through exchanging with practitioners about their research needs | Agree |
Preamble: IPPH is very committed to knowledge translation and has consistently embedded KT principles and requirements in many of its funding calls as well as contributed to other CIHR-led efforts. 1) Continue to actively foster the development of the six National Collaborating Centres for Public Health (NCCs), which emphasize knowledge synthesis, KT and network development, in support of improved public health policies, programs and practices (facilitated through a formal part-time secondment of the Scientific Director to PHAC) 2) Continue to integrate KT requirements in IPPH-led or co-led funding calls (e.g. IPPH's Research and Knowledge Exchange Initiative, KT Branch-led RFAs) 3) Strengthen knowledge translation capacity to support the effective use of evidence by policy makers and practitioners (e.g. joint Masters of Public Health Award Program call with PHAC) |
| 7. Performance Monitoring and Reporting - To ensure the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. | Agree |
Preamble: IPPH has demonstrated an ongoing, through its leadership and engagement in numerous systematic analyses (e.g. Theme 4 classification project; intervention research project referenced above; NHRDP data analysis) and evaluation activities of the organization. It supports the need for a strong CIHR Evaluation and Analysis Unit and notes that even the standard research classification system is under- developed for classifying PPH type research (as opposed to biomedical research). 1) Continue to undertake ongoing performance monitoring and reporting activities as achieved through: annual reports, annual operational plans and budgets, regular reporting to IAB, RPPC and Governing Council 2) Identify priorities for further analysis and evaluation to measure IPPH results - for e.g.
|
1Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.
2A request for applications (RFA) is the mechanism for encouraging applications for funding in a priority research area and/or strategic initiative.
Supplemental content (right column)
- Modified: