Mid-Term Evaluation of the Institute of Infection and Immunity (III)

Executive Summary

December 2005


Table of Contents


1. Executive Summary
    1.1 Overview of CIHR
    1.2 Overview of the Institute of Infection and Immunity (III)
        1.2.1 Mandate
        1.2.2 Vision and Mission
        1.2.3 Strategic Research Priority Themes
    1.3 Evaluation Objectives and Issues
        1.3.1 Methodology
2. Evaluation Results
    2.1 Relevance
    2.2 Effectiveness
    2.3 Delivery
3. Recommendations
4. Management Response

1. Executive Summary


1.1 Overview of CIHR

The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.

CIHR was created under The CIHR Act that came into force on June 7, 2000.

Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).

In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:

  • outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
  • outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
  • transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.

These strategic outcomes will be enabled through:

  • effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
  • organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.

CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.

CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and social, cultural, environmental and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.

Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.

CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).

Return to top

1.2 Overview of the Institute of Infection and Immunity (III)

1.2.1 Mandate

The Institute of Infection and Immunity (III) was established to support research in the enhancement of immune-mediated health and the reduction of the burden of infectious disease, immune-mediated disease, and allergy through prevention strategies, screening, diagnosis, treatment, support systems, and palliation, with a view to reducing the global burden of infection and immune-based disease and improve quality of life.

1.2.2 Vision and Mission

The vision of the III is to achieve national and global leadership by strongly supporting research in health and public policy on infectious disease control, and on the harnessing of healthy immune responses to protect against or mitigate health challenges.

Its mission is to establish national leadership, priorities and programs that promote innovative research to reduce the global burden of infection and immune-based diseases and improve the quality of life.

1.2.3 Strategic Research Priority Themes

In 2001, the III identified a number of priority research themes in the two sub-domains of the Institute (infectious diseases and host response), and, for each, set goals, many of which it has since attained. The research themes are the following:

  • Infectious Diseases:
    • Microbiologically Safe Food and Water: Develop a national strategy for research leading to microbiologically safe food and water;
    • Antimicrobial Resistance: Establish a national strategy for research into biomedical, genetic, epidemiological, health care, and socioeconomic practices that cause antimicrobial resistance in microbes;
    • HIV/AIDS and Hepatitis C: Establish a national HIV/AIDS research advisory partnership for research into HIV prevention, treatment and care at the biomedical, behavioural and social level; establish research programs that fill specific gaps in research into Hepatitis C and blood-borne infections at the biomedical, social, and behavioural level;
    • Novel Vaccine Development: Establish research into new vaccine targets, formulations and delivery systems for immunization against infectious disease and support research to address efficacy and ethical issues in the use of vaccines; and
    • Emerging Infectious Diseases: Support collaborative and innovative health research into new emerging infectious diseases, public health concerns, and global threats to health.
  • Host Response:
    • Organ Transplantation and Regeneration: Support research on factors underlying acute and chronic graft rejection, organ regeneration using stem cells, and their social and ethical impact;
    • Autoimmune Diseases: Promote interdisciplinary research programs focused on understanding the common mechanisms that cause autoimmune diseases;
    • Asthma and Allergy: Support research into causes and treatment for allergy and asthma; and
    • Innate Immunity: Promote better understanding of innate immunity and its role in disease prevention.

Return to top

1.3 Evaluation Objectives and Issues

The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:

  • to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
  • to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.

The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:

  • Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
  • Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
  • Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
  • Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?

The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.

1.3.1 Methodology

The evaluation consisted of four main lines of evidence:

  • a review of documents and administrative data relating to the III;
  • 50 key informant interviews with III staff and IAB members, researchers and students, stakeholders and partners;
  • case studies of two III initiatives; and
  • a telephone survey of 188 funded and 55 non-funded researchers affiliated with the III.1

Return to top

2. Evaluation Results


2.1 Relevance

There is general agreement across interviewees and survey respondents that the mandate and research priorities of III are appropriate. A small number of interviewees noted the mandate of III, although appropriate, is too broad given the Institute's budget.

The vast majority of interviewees feel that the research domains that III focuses on are appropriate. There is near consensus among interviewees that infection and immunity are complementary research domains, although some noted that traditionally there was only limited collaboration across infection and immunity. The Institute, through combining or bringing together these two research areas, may have resulted in increased collaboration.

Evidence from surveys and key informant interviews indicates that the Institute is seen by most as being the most appropriate mechanism for developing Canadian capacity, research excellence and knowledge translation. Some interviewees feel that III may not be the most appropriate mechanism for knowledge translation or may not be the most appropriate mechanism to take on knowledge translation on its own.

Interviewees see the advantages of the virtual model for the Institute as its flexibility and sustainability. A key challenge to the virtual model for III identified was succession planning and changes in the Scientific Director. A change in Scientific Director is likely to result in either the loss of existing staff or the need for staff to relocate to another part of Canada. The risk is that this will result in a loss of corporate memory. This challenge is not unique to III and will likely need to be addressed at a higher level within CIHR. As well some challenges with respect to communication were identified as a result of the geographic dispersion of Institute staff and IAB members. However, this challenge would likely exist even if the Institute were not virtual since little can be done about the geographic dispersion of IAB members while still maintaining the quality and diversity of IAB membership.

Return to top

2.2 Effectiveness

There is general agreement that III has made a contribution to the creation of new knowledge in the context of CIHR's overall objective. However, there is a strong sense among many interviewed that the contribution of III to the creation of new knowledge will increase over time as projects funded by III produce results/new knowledge. Specific areas where interviewees noted that new knowledge had been created were in the areas of SARS and the Safe Food and Water Initiative.

Slightly over half of those interviewed feel that knowledge translation has been a priority for III. While a definition of Knowledge Translation has been published on the CIHR Web site for several years and a five year Knowledge Translation strategy was approved by the CIHR Governing Council in June 2004, the key reasons cited by those who do not feel knowledge translation has been a priority are the lack of direction from CIHR corporate with respect to the definition of knowledge translation and the lack of a clear definition of the roles and responsibilities of Institutes with respect to knowledge translation.

Although only about half of those interviewed feel knowledge translation has been a priority, most feel that III has made a contribution. This apparent inconsistency is explained by the fact that many of those interviewed were unable to assess the internal priorities of III but were able to assess the contributions made by the Institute. So although many interviewees did not know whether or not III treated knowledge translation as a priority, they were able to assess the impacts of III on knowledge translation. The survey of funded researchers found that III is seen to have made a contribution to knowledge translation by 76% of respondents; these survey results for III are similar to those for the CIHR overall.

Evidence suggests that III has been successful in building and maintaining capacity within the infection and immunity research communities. The main mechanisms through which the Institute has done this are through funding to 14 training centres directly funding trainees each year and the provision of "bridging grants".

In addition to developing human capacity, CIHR Institutes are also expected to work to develop infrastructure capacity in their respective research domains. III has undertaken to develop a SARS patient sample database. However, little has been undertaken with respect to infrastructure development beyond that. We note that the infection and immunity research communities were relatively well developed prior to III and therefore a significant research infrastructure pre-existed III.

There is evidence to suggest that III has made progress in contributing to the transformative vision of CIHR. Interviewees, largely representing staff and IAB members, noted the transformative nature of the SARS Consortium and the Safe Food and Water Initiative that included researchers from a range of disciplines.

Although there is quantitative and qualitative evidence to suggest that III has made a contribution to the transformative vision of CIHR, there is also some evidence to suggest there are some who continue to resist the interdisciplinary nature of CIHR. There are some researchers who continue to feel that the switch from the Medical Research Council (MRC) to CIHR has resulted in less funding being available to bench research. Funding data for MRC and CIHR refute this view. Some interviewees noted that although III has made progress in including research within Themes three and four, this has presented some challenges because infection and immunity research is traditionally focused on Themes one and two.

Evidence from interviews and the survey suggests that III has performed well with respect to achieving its mandate and research priorities. This is not to say that III has met or fulfilled its mandate and priorities but the evidence suggests that III is moving in the right direction. While there has been significant activity in areas of the Institute's priorities, there is less success on overall CIHR priorities of ethics and knowledge translation. There is currently little data beyond qualitative and anecdotal evidence to assess whether III has made progress. In addition, there are no performance targets yet established that would more effectively enable the measurement of success.

Based on data provided by the Institute in its Performance Report, III has formed over 40 partnerships, linkages, alliances, and exchanges with a broad range of organizations including federal and provincial governments, research organizations, non-governmental organizations and international organizations. Interviewees and survey respondents generally believe that III has been successful in developing partnerships and facilitating collaboration.

One method used by Institutes to influence research direction is the creation of funding opportunities in the areas of strategic and emerging research importance relevant to their research domains. The extent to which these opportunities generate a response from the research community is an indication of the level of activity and interest in these research areas. Data on the number of applications submitted in response to each of the Institute's programs indicate a good level of interest within the research community. Interviewees generally agree that III has been successful with respect to influencing the research agendas within the infection and immunity communities. However, many noted that more time and funding is necessary for III to fully realize this goal.

Return to top

2.3 Delivery

The Institute generally relies on IAB members to provide guidance in strategic and operational planning. IAB members are seen as being representatives of the researcher and stakeholder communities and thus credible intermediaries between the Institute and the researcher and stakeholder communities. The in-person IAB meetings held three times per year are seen as being the most productive and useful with respect to planning. Meetings held via teleconference are seen as being less productive with respect to planning activities. In addition to input from the IAB, the Scientific Director travels frequently across Canada to obtain input and feedback from researchers and stakeholders, although stakeholders who were interviewed were split on whether they were involved in the planning process.

Interviews and results from the survey of funded and non-funded researchers indicate that most feel that III has done a good job of identifying emerging priorities within its mandate.

Staff and IAB members interviewed are satisfied with the functioning and contribution of the IAB. The only challenge identified by a small number of interviewees is the tendency for some IAB members to lobby on behalf of their area of research. We note that this may be the result of IAB members being selected from different areas of research and so there is a sense that they are on the III IAB as representatives of their area of research. Although this was identified as a challenge, there is no evidence that this has impeded the functioning of the IAB.

III seeks to maintain visibility within the infection and immunity research communities through regular participation at national and international meetings related to the Institute's mandate. These meetings are seen to be important by Institute staff because they are vehicles for knowledge exchange and lead to the development of new research initiatives. IAB members are again seen as a key link to the research and stakeholder communities. An important method of informal consultation with the communities occurs at in-person IAB meetings held three times per year. A dinner is held as part of each of these meetings and III invites members of the local research community and stakeholder representatives to attend. This has resulted in significant in-person contact with local researcher and stakeholder communities and III staff and IAB members.

The Institute uses a variety of mechanisms through which to communicate with key public audiences, including researchers, stakeholders and the general public. The researcher survey showed 95% of funded and 94% of non-funded researchers were familiar with III at least to some extent. These mechanisms include developing communications documents, presentations and media relations. Although III has directed some communications directly to the public, these communications are not generally seen as having had a significant impact on the general public.

There is some evidence to suggest that collaboration with respect to communication between III and CIHR corporate could be improved. This evaluation found some evidence of dissatisfaction in the relationship between III and CIHR corporate. Specific areas where some staff and IAB members feel the relationship could be improved include: the peer review process; the integration of programs; communication and interaction of Ottawa-based staff of the 13 Institutes; and clarifying the role of CIHR corporate in relation to the Institutes.

There is agreement among interviewees that funding available to III is insufficient relative to its mandate. Some suggested that the Institute adapt its mandate to reflect its budget, however the mandate and the research priorities of III reflect, or are linked to, the overall objectives and mandate of the CIHR. A paring down of the III mandate and priorities implies that the CIHR could not meet its objectives with respect to research in infection and immunity.

The evaluation found some evidence of dissatisfaction with the inclusion of social science-based research within the CIHR and III mandate. This dissatisfaction stems from a misunderstanding over the amount of funding going towards Themes one and two within CIHR relative to the funding available previously through the MRC. Some researchers interviewed feel that the CIHR focus on the four research themes has resulted in less funding to biomedical and clinical research. Funding data for CIHR and MRC indicate, however, that this is not true. In fact, total funding available to biomedical and clinical research has increased relative to what was available under the MRC.

Return to top

3. Recommendations

Following are recommendations that emerge from this mid-term evaluation of III. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.

Recommendation 1:

The Institute has been found to be doing well and is encouraged to continue the following:

  1. sustain its efforts in the areas of research excellence, capacity development and funding strategic priorities;
  2. maintain the current structure and operation of the IAB;
  3. sustain its current efforts in collaboration and partnering;
  4. sustain efforts in knowledge creation with an increased focus on the immunity research community;
  5. sustain its current knowledge translation activities; and
  6. sustain its contributions in the area of ethics.

The following areas are ones in which it is recommended the Institute take action to improve:

Recommendation 2:

Communication - While the Institute's activities have, in some years, received significant media attention, the Institute should - with the support of CIHR corporate - assess mechanisms for raising its profile within its community and the general public by drawing attention to its activities and results.

Recommendation 3:

Planning and Consultation Mechanisms - While overall planning and consultation mechanisms have been found to be generally effective, the Institute should assess whether there is a need to more actively seek input from stakeholders (i.e. partners) and, if necessary how best to seek input from stakeholders (i.e. options for improving the response rate to future surveys since the survey mechanism allows for more broad based engagement of the researcher and stakeholder communities).

Recommendation 4:

Knowledge Translation - The III should review its efforts in support of KT. The efforts expended by the III in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the III are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.

Recommendation 5:

Contribution to Transformative Vision - While overall efforts have been made in the implementation of the transformative vision of CIHR which III is encouraged to sustain, the Institute - with the support of CIHR corporate - should communicate the benefits and funding impacts of CIHR and its transformative vision to researchers.

Recommendation 6:

Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.

Return to top

4. Management Response

Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:

Recommendation Response Action Plan
1. Consultation and Communication - While the Institute's activities have, in some years, received significant media attention, the Institute should - with the support of CIHR corporate - assess mechanisms for raising its profile within its community and the general public by drawing attention to its activities and results. Agree The Institute, in conjunction with the CIHR Corporate Affairs, Marketing and Communication Branch, is developing a communication plan that targets key audiences (researchers, stakeholders, government and the public through media channels). The plan will build on existing Institute communication vehicles (Web site, newsletter, email distribution list) and will look for opportunities and mechanisms to inform a broader audience.
2. Planning and Consultation Mechanisms - While overall planning and consultation mechanisms have been found to be generally effective, the Institute should assess whether there is a need to more actively seek input from stakeholders (i.e. partners) and, if necessary how best to seek input from stakeholders (i.e. options for improving the response rate to future surveys since the survey mechanism allows for more broad based engagement of the researcher and stakeholder communities). Agree The organization and structure of the second round of strategic planning for the Institute is currently under development. Input from stakeholders will be actively sought during this process. Mechanisms for longer term engagement of the diverse elements of the infection and immunity community, such as consultations during annual meetings of professional societies and health charities, will be discussed as part of the process.
3. Knowledge Translation - The III should review its efforts in support of KT. The efforts expended by the III in the area of KT do not appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the III are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT. Agree Defining the scope and content of knowledge translation is an ongoing challenge. The Institute has expended a major ongoing effort to be proactively involved in the determination of national and international policy and outcomes relating to its mandate in infection and immunity, particularly in its ongoing relationship with the PHAC. The Institute will continue to engage other government agencies, professional societies, voluntary health organizations and key national and international organizations to build upon an established repertoire of activities that will advance the transfer and uptake of knowledge and impact the formulation of policy aimed at improving the health of Canadians. The Institute will highlight its KT activities in its communications plan so that IAB members, researchers and stakeholders are aware of these activities.
4. Contribution to Transformative Vision - While overall efforts have been made in the implementation of the transformative vision of CIHR which III is encouraged to sustain, the Institute - with the support of CIHR corporate - should communicate the benefits and funding impacts of CIHR and its transformative vision to researchers. Agree

The Institute will continue to develop programs which emphasize the enhanced potential to be obtained from strategic and multi-disciplinary collaborative research programs. In promoting these initiatives, the Institute will profile the goals and anticipated outcomes from the strategies outlined in the programs.
Definition of the transformative vision will also be included in the communications plan as this term is also not well defined or understood by the audiences surveyed during the research and key informant interviews.

The Institute will work with the CIHR Corporate Affairs - Evaluation and Analysis Unit to establish better mechanisms for tracking contributions to the four CIHR themes.

5. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. Agree The Institute is acutely aware of the need to systematically evaluate the performance and outcomes of Institute activities and programs and has committed significant personnel resources for the mid-term evaluation and international review of CIHR. A continued effort will be made to work with the CIHR Corporate Affairs Evaluation and Analysis Unit to refine the common indicators and develop a process for ongoing performance monitoring. The Institute will collect performance data from Institute-funded research programs. This data will be used by the Institute and IAB to refine future initiatives and assess research outcomes and impacts.


1 Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.

Supplemental content (right column)