Mid-Term Evaluation of the Institute of Nutrition, Metabolism and Diabetes (INMD)
Executive Summary
December 2005
Table of Contents
1. Executive Summary
1.1 Overview of CIHR
1.2 Overview of the Institute of Nutrition, Metabolism and Diabetes (INMD)
1.2.1 Mandate
1.2.2 Strategic Research Priority
1.3 Evaluation Objectives and Issues
1.3.1 Methodology
2. Evaluation Results
2.1 Relevance
2.2 Effectiveness
2.3 Delivery
3. Recommendations
4. Management Response
1. Executive Summary
1.1 Overview of CIHR
The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
CIHR was created under The CIHR Act that came into force on June 7, 2000.
Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).
In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:
- outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
- outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
- transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.
These strategic outcomes will be enabled through:
- effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
- organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.
CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.
CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and social, cultural, environmental and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.
Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.
CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).
1.2 Overview of the Institute of Nutrition, Metabolism and Diabetes (INMD)
This Institute supports research on enhancing health, as it pertains to diet, digestion, excretion, and metabolism. It aims to focus on addressing causes, prevention, screening, diagnosis, treatment, support systems, and palliation for a wide range of conditions and problems associated with hormone, digestive system, kidney and liver function.
1.2.2 Strategic Research Priority
The INMD identified a single priority/Strategic Initiative (SI) - Excellence, Innovation and Advancement in the Study of Obesity and Healthy Body Weight - after extensive consultation with its constituency. The focus on obesity and healthy body weight is justified in large physical, psychological and economic costs associated with obesity and the multiple and interacting causes of it and difficulties in preventing and treating the problem. Projects funded by the INMD range from efforts aimed at unravelling the complex biological mechanisms regulating body weight to projects examining individual behavioural responses to environmental stimuli.
1.3 Evaluation Objectives and Issues
The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:
- to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
- to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.
The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:
- Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
- Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
- Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
- Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?
The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.
The evaluation consisted of four main lines of evidence:
- a review of documents and administrative data relating to the INMD;
- forty key informant interviews with INMD staff and IAB members, researchers and students, stakeholders and partners, as well as a focus group discussion with IAB members (held at an IAB meeting);
- case studies of three INMD initiatives; and
- a telephone survey of 168 funded and 43 non-funded researchers affiliated with the INMD.1
2. Evaluation Results
2.1 Relevance
The evaluation evidence indicates that the mandate of the INMD is appropriate and relevant. A majority of key informants in all respondent groups regard the Institute's mandate as appropriate (though very broad) and almost two-thirds of funded INMD researchers and two-fifths of non-funded researchers see a close fit of their research interests with the Institute's mandate. The majority of key informants also regard the research domains within the INMD's mandate as appropriate and do not suggest any changes at this time, though some feel there are areas that do not fit as well, such as gastrointestinal (GI) and liver disease.
In addition, the single strategic priority of Obesity/Healthy Body Weight (OHBW) is widely viewed as appropriate and relevant by key informants. Awareness of this strategic priority is very high — among surveyed researchers who felt their Institute had been quite successful in identifying key priorities, correct identification of the INMD's priority was considerably higher than was observed for other CIHR Institutes. Key informants express some concern, however, that some research communities that are not highly involved in OHBW but still fall within the broader mandate may have received insufficient attention to date (e.g., gastroenterology, liver and hormone/endocrinology researchers).
It is difficult based on the available evidence to draw firm conclusions on the extent to which disenfranchised research communities present a significant problem that INMD needs to address at this time. This is, in part, because a specific question on feeling marginalized due to the Institute's selection of a single strategic initiative was not included in the researcher survey. As noted above, however, some indication that this may be an issue is provided by the survey findings — a majority of funded INMD researchers but only two-fifths of non-funded INMD researchers (compared to one-half of non-funded researchers overall) indicated a close fit of their research interests with the Institute's mandate. This being said, there is considerable evidence to support the choice of a single strategic priority.
INMD is generally viewed as an appropriate mechanism to make a difference in its research domain. Moreover, researchers surveyed believe there is a need for the INMD to support research excellence, capacity development and (to a slightly lesser extent) funding of strategic priorities. In the view of a minority of key informants, however, INMD may not have the capacity/resources to fully achieve all three of its intended outcomes — developing Canadian capacity, research excellence and knowledge translation — and the Institute may not be the most appropriate organization to take the lead role in knowledge translation (e.g., because major health charities/NGOs have a key role to play). No alternatives to the INMD mechanism were identified, although some improvements were suggested (e.g., broadening linkages and partnerships). Also, the CIHR model of virtual Institutes is generally regarded as appropriate, though some suggest that implementation of this model could be refined, for example, with more standardization and resources and as well as measures to ease disruption of the upcoming SD/Institute transition.
2.2 Effectiveness
Overall, INMD has performed well and has made good progress in contributing to CIHR's overall objective and in fulfilling the Institute's mandate and strategic priority.
Through RFAs2 and the funding of research, INMD is making progress in creating new knowledge, in particular, in the area of Obesity/Healthy Body Weight. Most funded INMD researchers agree that the Institute has contributed in advancing health research to respond to opportunities/priorities and create health knowledge. As well, case studies of one-year funding mechanisms (Pilot Project Grants and Planning and Development Grants), Canada on the Move (COTM) and the Chronic Disease New Emerging Team (NET) partnership indicate that the INMD is helping to stimulate research on OHBW and create new knowledge. Although most key informants also believe that progress is being made toward knowledge creation, many feel that it is premature to conclude how much new knowledge will truly be created as a direct result of INMD's efforts/funding.
INMD is devoting effort toward and funding knowledge translation and related activities, such as meetings, workshops/dialogues, syntheses and public engagement. Most funded INMD researchers surveyed believe that the Institute's activities support KT to at least some extent and researchers are highly involved in disseminating their research findings. Case studies provide further evidence of significant KT activities, including dissemination of research results (e.g., papers, publications and presentations), bringing together researchers and policy makers, and public engagement. Many key informants agree that INMD has done important groundwork related to knowledge translation, however, they generally believe that the application of research knowledge to develop/change policies or improve services/clinical practices has not yet occurred and some note that it is premature to observe these sorts of longer-term impacts. Perceived challenges are the general lack of understanding of KT as well as a lack of clear guidance from CIHR central as to what Institutes are expected to do.
INMD has helped to build research capacity through the utilization of a range of grants and awards to fund programs and through significant investments in STIHRs and Training Awards, which have been increasing annually. Key informants generally agree that INMD has made a strong contribution to capacity development. Similarly, the majority of funded researchers in the survey feel that INMD has contributed, to at least some extent, to the development of people and the research environment. In addition, case studies indicate that INMD initiatives (e.g., one-year funding mechanisms and the Chronic Disease NET) are contributing to capacity development, for example, by attracting trainees and establishing linkages among researchers from different disciplines.
INMD is contributing to CIHR's transformative vision through its fostering of interdisciplinary research and multi-sector partnerships, its focus on a cross-cutting strategic priority (OHBW) with potential to improve the health of Canadians, and its innovative initiatives such as COTM that have a strong KT element. Moreover, in the survey, funded INMD researchers report that an average of 61% of their projects are interdisciplinary. Most of the CIHR funding for research related to the INMD's mandate is in Theme 1 and, to a lesser extent, Theme 2; only a small proportion of funding is in Themes 3 and 4. Obesity grants and awards issued directly by INMD do however place more emphasis on Theme 4.
The Institute has also made a modest contribution to the CIHR ethics mandate, for example, by contributing financially to joint strategic initiatives with some relation to ethics issues and dealing with ethical issues associated with private sector partnerships for COTM. There is low awareness of the ethics mandate and the Institute's contribution in this regard, however, among many key informants.
Interviewees generally feel that INMD is performing well relative to its mandate and, in particular, its strategic priority in Obesity/Healthy Body Weight. The Institute's progress is confirmed by its significant investments in RFAs related to OHBW and also some other areas of its broader mandate. Similarly, the majority of funded INMD researchers perceive that the mandate and strategic priority have been achieved to at least some extent. CIHR investments in research related to nutrition, metabolism and diabetes have increased steadily over the past five years, further indicating that INMD is influencing research under its mandate.
Key informant interviews, case studies (e.g., of COTM and the Chronic Disease NET) and the document review indicate that INMD has collaborated extensively and developed numerous partnerships, linkages and alliances with a broad range of stakeholders from numerous sectors. The Institute's Performance Report lists approximately 200 organizations with which INMD has some form of linkage, alliance or public engagement, including departments/agencies from all levels of government, health charities, private foundations, non-profit NGOs, research institutes and private industry/industry associations. INMD has also worked effectively with other CIHR Institutes and has received financial contributions totalling roughly $4.5M for INMD-led RFAs from eight Institutes (ICRH, IGH, IHDCYH, III, IPPH, ICR, IG, and IAPH). These contributions have been primarily for initiatives related to OHBW, but have also included some other areas such as NET grants on chronic disease and Type 1 diabetes. Potential barriers to partnership development, in the view of key informants, include a lack of policies at CIHR central (particularly in the initial years of the organization), lack of resources, some resistance to partnering with private industry, and the time required to develop and nurture partnerships.
INMD has demonstrated leadership and has had an influence on research agendas, in particular, in the area of Obesity/Healthy Body Weight. For example, as noted in the Performance Report, there has been growth in applications to repeat offerings under the Priority Announcement3 for operating grants and the Pilot Project Grant under OHBW, which indicates some influence of the strategic initiative. Key informants agree that the Institute has demonstrated leadership and influence by focusing attention and research funding on OHBW, and also by encouraging interdisciplinary research and partnering strategically. Leadership is also illustrated by innovative initiatives such as COTM.
2.3 Delivery
INMD has used strategic planning mechanisms effectively. The initial strategic plan and selection of the key area of strategic research importance — Obesity/Healthy Body Weight - were based on extensive consultations (i.e., an environmental scan of a wide range of stakeholders and researchers representing all four themes/pillars) as well as the advice of the IAB. Key informants generally believe that these mechanisms have been effective. Moreover, most researchers surveyed feel that the INMD has successfully identified and responded to emerging priorities within its mandate, to at least some extent. While planning at the operational level was challenging in the Institute's early years (e.g., due to staff turnover), improvements have been made.
The IAB has been operating effectively and contributing to the achievements of the Institute (e.g., by providing useful input into the direction and strategic priority of the Institute and for key initiatives such as COTM). Staff and Board members are highly satisfied with the operation, membership and effectiveness of the Board.
INMD has consulted extensively and effectively with a wide range of stakeholders using both formal and informal methods. Key informants and researchers surveyed feel that the INMD has been responsive to input and feedback. In addition, the Institute has utilized a wide range of communications approaches/vehicles and has received significant media coverage (e.g., for COTM and obesity issues). Some key informants feel, however, that the communication of INMD's initiatives and achievements could be improved.
Factors that exert some influence on the effectiveness of INMD include central CIHR functions and funding levels. Following an initial lack of responsiveness of central CIHR functions, the degree of central support for INMD has improved over the years though some deficiencies remain. In particular, there is perceived to be a lack of consistent central leadership and guidance on KT and, for a minority of key informants, ethics. Most key informants view the level of funding ($1M for the Institute support grant and approximately $6.5M for strategic funding) as inadequate to make a difference in all areas covered in INMD's broad mandate. Although mixed views are expressed on the 70:30 split between investigator-initiated and strategic research funding, a slight majority regards this balance as appropriate.
3. Recommendations
Following are recommendations that emerge from this mid-term evaluation of INMD. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.
Recommendation 1:
The Institute has been found to be doing well and is encouraged to continue the following:
- support for research excellence, capacity development and funding strategic priorities, in partnership with other CIHR Institutes and stakeholders (e.g., government policy makers, NGOs/health charities, the private sector);
- maintain the current structure and operation of the IAB continuing to ensure an appropriate balance and membership;
- maintain its planning mechanisms;
- continue its substantial efforts in consulting with researchers and a diversity of stakeholders using both formal and informal methods;
- continue its work in collaborating and developing partnerships with a range of organizations and sectors, as this has been a key strength; and
- continue its efforts in contributing to the transformative vision of CIHR through its emphasis on interdisciplinary research, multi-sectoral partnerships and its innovative initiatives such as COTM that incorporate a focus on translating knowledge and improving the health of Canadians.
The following areas are ones in which it is recommended the Institute take action to improve:
Recommendation 2:
Strategic Priority - INMD made a risky and far-sighted decision in choosing to concentrate their efforts on a single strategic priority. It is too early to tell if the choice to pursue a single strategic priority is yielding new knowledge that will lead ultimately to a reduction in obesity-related health problems as well as obesity. It is recommended that INMD and CIHR should carefully monitor the results of the implementation of this approach to determine its overall effectiveness.
Recommendation 3:
Communications - In its communications, it is recommended that INMD should endeavour to reach out to the full range of researchers and stakeholders in its community, including, for example, researchers that may feel disenfranchised and those from fields such as the social sciences that may not traditionally have done obesity-related research. It is important to educate all researchers/stakeholders about the Institute's mandate and strategic priority (and the distinction between the two) as well as the opportunities available to them either directly through the Institute.
Recommendation 4:
Knowledge Creation - It is recommended that the Institute continue its efforts in creating knowledge on obesity/healthy body weight and, to the extent feasible with available resources, look for opportunities to support further knowledge creation in other domains within its mandate beyond the single strategic priority of obesity/healthy body weight.
Recommendation 5:
Knowledge Translation - Knowledge translation is an area for improvement across CIHR including all Institutes. It is recommended that the Institute should continue to define its roles and responsibilities and to improve knowledge translation within its own sphere of influence and effectively communicate the results.
Recommendation 6:
Contribution to Ethics Mandate - It is recommended that the Institute continue to contribute to the implementation of CIHR's overall ethics mandate and ensure adequate communication of the activities and results to relevant audiences and stakeholder groups.
Recommendation 7:
Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.
4. Management Response
Overall comments on the report, including, if desired, comment on Recommendation 1 that suggests continuing certain activities that are going well:
In general, INMD finds the evaluation report useful, although there were no real surprises contained in the data and report.
In general the report is clear, but there are places where we would have liked additional information regarding the context of the specific data used to make comments or recommendations. It is sometimes difficult to determine whether the comments are based on views expressed by many or are the dissenting voices expressed by only a few respondents. This is especially important given that the key informant interviews included only 8 researchers with an unknown distribution across the many areas in the INMD mandate.
As a result of the limited number of key informant researcher interviews, the views or our research community were primarily collected via the researcher survey. It is clear from the data in the survey that most researchers surveyed in this manner did not distinguish between an Institute's mandate and its strategic priority or priorities. So when the surveyed researchers say that their research is not a close fit with the Institute mandate, they may be confusing the mandate with the priority and in the case of INMD they may be reporting that they are not working in the area of obesity. The interpretation of the results from the researcher survey must take into account this lack of distinction by the research community.
As noted in the evaluation, INMD's decision to select a single strategic priority also makes it an easy target for those who feel left out because they do not work on the problem of obesity and have no interest in applying their skills in this area. While this has undoubtedly led to some feelings of disenfranchisement, it is unclear how widespread this feeling is, whether it is any more pervasive for INMD as compared to disenfranchisement of researchers from other institutes, whether selection of multiple priorities would have lead to less disenfranchisement, or whether it can be managed through improved communications. It is clear that some communities that could easily have been unhappy with INMD's decision (e.g. kidney) have an excellent relationship with the Institute. This is demonstrated by the recent presentation of a partnership award to INMD from the Kidney Foundation of Canada for our work with KFOC and the Canadian Society of Nephrology on the KRESCENT training program.
It is also important to note the while some have expressed concern, many others have confirmed that INMD's decision to focus our limited resources on a single major priority was the most strategic decision possible. Clearly, this has resulted in considerable attention from researchers, policymakers and the general public. Early signs of impact indicate an 8-fold increase in the number of applicants to our obesity research programs and a 5-fold increase in the number of funded applications. This is well above the increase expected by the doubling of the CIHR budget and the targeting of INMD's strategic funds in this area. INMD is also one of the Institutes with the highest media profile.
The institute is proud of what we have been able to achieve and look forward to continuing to improve our efforts in all areas, including those where the evaluation has identified our strengths.
| Recommendation | Response | Action Plan |
| 1. Strategic Priority - INMD made a risky and far-sighted decision in choosing to concentrate their efforts on a single strategic priority. It is too early to tell if the choice to pursue a single strategic priority is yielding new knowledge that will lead ultimately to a reduction in obesity-related health problems as well as obesity. It is recommended that INMD and CIHR should carefully monitor the results of the implementation of this approach to determine its overall effectiveness. | Agree | As noted above, INMD has begun to assess the impact of its decision by the examination of early outcomes such as the number of applicants and the number of applications (funded and unfunded) to CIHR in the area of obesity and healthy body weight. Search parameters are being validated and data is being compared to other areas in the mandate that were not identified as a priority. INMD is also conducting an environmental scan of other federal and provincial funding agencies in partnership with the Heart and Stroke foundation of Canada to obtain a more detailed picture of obesity research in Canada. This scan includes key informant interviews with researchers, University administrators and policy makers to assess our impact on programs of research and knowledge transfer. |
| 2. Communications - In its communications, it is recommended that INMD should endeavour to reach out to the full range of researchers and stakeholders in its community, including, for example, researchers that may feel disenfranchised and those from fields such as the social sciences that may not traditionally have done obesity-related research. It is important to educate all researchers/stakeholders about the Institute's mandate and strategic priority (and the distinction between the two) as well as the opportunities available to them either directly through the Institute. | Agree |
INMD will continue its ongoing effort to communicate with all of our research communities and stakeholders in an effort to educate them about CIHR, INMD activities and the distinctions between research priorities and the Institute mandate. We will use multiple vehicles for this communication, including:
|
| 3. Knowledge Creation - It is recommended that the Institute continue its efforts in creating knowledge on obesity/healthy body weight and, to the extent feasible with available resources, look for opportunities to support further knowledge creation in other domains within its mandate beyond the single strategic priority of obesity/healthy body weight. | Agree |
Although INMD has identified a single strategic priority, it has always operated on the principle that strategic funds are allocated to the strategic priority and to strategic partnerships in other areas of the mandate. As a result, nearly half the strategic funds have been allocated to research in other areas of the mandate. Funding programs either developed or contributed to in areas outside our strategic priority include Chronic Disease New Emerging Teams, the KRESCENT Kidney research training program, Diabetes Doctoral awards and Strategic Training programs in Health Research. We will continue to use this strategy of focusing our funds on our priority and on strategic partnerships with other organizations including other government and non-governmental organizations. Institute Support Grant funds will also continue to be used to help develop research in all areas of the INMD mandate. |
| 4. Knowledge Translation - Knowledge translation is an area for improvement across CIHR including all Institutes. It is recommended that the Institute should continue to define its roles and responsibilities and to improve knowledge translation within its own sphere of influence and effectively communicate the results. | Agree |
INMD will continue to work with CIHR to help define CIHR's role in the KT arena. Using our strategic priority of obesity and healthy body weight as a working example, we will continue to work with other organizations such as Health Canada, the Public Health Agency, the BC Provincial Health Services Authority, the Chronic Disease Prevention Alliance of Canada, and many others to define our respective roles in KT. INMD is launching a 'Knowledge Synthesis and Translation' RFA using the Institute Support Grant to support projects that disseminate health research, through synthesis, scans or translation activities- a yearly competition. We expect to fund five projects at $10,000 each per year. |
| 5. Contribution to Ethics Mandate - It is recommended that the Institute continue to contribute to the implementation of CIHR's overall ethics mandate and ensure adequate communication of activities and results to relevant audiences and stakeholder groups. | Agree |
INMD and the IAB will dedicate more time to discussing how INMD could most effectively contribute to the CIHR ethics mandate. The IAB and SD will identify issues that are relevant to INMD's strategic priority and mandate. INMD will include a session on research ethics at its upcoming capacity building meeting for teams and new obesity researchers. |
| 6. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. | Agree |
As noted above, INMD is undertaking a substantial effort to determine the outputs and outcomes of its focus on obesity. In addition, effort will be made to monitor other areas of the INMD mandate. Some specific programs have already been evaluated such as our obesity peer review panel and our 1-year funding programs. Additional evaluations will be undertaken as programs mature. |
1 Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.
2 A Request for Applications (RFA) is the mechanism for encouraging applications for funding in a priority research area and/or strategic initiative.
3 A priority announcement is a mechanism used to fund highly rated research applications that are determined to be relevant to CIHR's research priority areas and do not receive funding through CIHR's regular competitions.
Supplemental content (right column)
- Modified: