Mid-Term Evaluation of the Institute of Musculoskeletal Health and Arthritis (IMHA)
Executive Summary
December 2005
Table of Contents
1. Executive Summary
1.1 Overview of CIHR
1.2 Overview of the Institute of Musculoskeletal Health and Arthritis (IMHA)
1.2.1 Vision
1.2.2 Mission
1.2.3 Research Priorities
1.3 Evaluation Objectives and Issues
1.3.1 Methodology
2. Evaluation Results
2.1 Relevance
2.2 Effectiveness
2.3 Delivery
3. Recommendations
4. Management Response
1. Executive Summary
1.1 Overview of CIHR
The Canadian Institutes of Health Research (CIHR) is the major federal agency responsible for funding health research in Canada. It aims to excel in the creation of new health knowledge, and to translate that knowledge from the research setting into real world applications. The results are improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
CIHR was created under The CIHR Act that came into force on June 7, 2000.
Its mandate is to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system (Bill C-13, April 13, 2000).
In pursuit of its mandate and vision, CIHR has articulated the following five expected outcomes, three of which are strategic and the other two, enabling:
- outstanding research: to advance health knowledge, through excellent and ethical research, across disciplines, sectors, and geography;
- outstanding researchers in innovative environments: to develop and sustain Canada's health researchers in vibrant, innovative and stable research environments; and
- transforming health research into action: to catalyze health innovation in order to strengthen the health system and contribute to the growth of Canada's economy.
These strategic outcomes will be enabled through:
- effective partnerships and public engagement: to engage with the public through meaningful dialogue and establish effective partnerships with key stakeholders; and
- organizational excellence: to achieve its mandate through excellence in staff, service delivery, systems, and management.
CIHR emphasizes multidisciplinary approaches to addressing health problems. The approach includes advancing research in four areas (or themes): biomedical, clinical, health systems and services, and the health of populations, societal and cultural dimensions of health and environmental influences on health.
CIHR's mandate and structure are unique in the world. CIHR is structured around 13 virtual geographically distributed Institutes that each support research in biomedical, clinical, health systems and services and population health. The Institutes are based in universities or teaching hospitals across the country, but may also have staff located in a variety of other venues. The Institutes are part of a larger national research network that links researchers and other stakeholders across the country.
Each Institute is headed by a Scientific Director who is an internationally recognized leader in his or her field and has on average five or six dedicated staff members. Scientific Directors receive guidance from their respective Institute Advisory Boards (IABs), made up of volunteers from all areas of the health research community, including those who fund research, those who carry it out and those who use its results. The Institutes are formally accountable to the CIHR President, the CIHR Governing Council and, through the Minister of Health, to Parliament.
CIHR's research funding for 2004-05 was $619M (up from $275M in 1999-2000). Total expenditures including administration were $666M in 2004-05, compared to $289M in 1999-2000. In 2004-05, $84M was allocated to Institutes to fund strategic research and $13M in support funding. Funds for strategic research within CIHR (including strategic research funded by the Institutes and by CIHR) represent about 30% of overall research funds (the remainder is allocated through the CIHR open competitions).
1.2 Overview of the Institute of Musculoskeletal Health and Arthritis (IMHA)
IMHA's vision is to sustain health and enhance quality of life by eradicating the pain, suffering and disability caused by arthritis, musculoskeletal, oral and skin conditions.
IMHA's mission reflects that of CIHR — to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge in all areas relevant to arthritis, rehabilitation, bone, muscle, skin and oral health and to translate that new knowledge into improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.
To achieve its mission, the IMHA has identified the following research priorities:
- Physical Activity, Mobility and Health: A healthy musculoskeletal system is fundamentally linked to an individual's ability to be mobile and physically active. More to the point, however, is the fact that personal well being, and the health of many if not all other body systems, is negatively impacted when musculoskeletal health is impaired. While these are commonly accepted principles, the real challenge for researchers is to take this general knowledge one step further. In other words, to conduct investigations that will help us better understand the complex relationships between physical activity, mobility, musculoskeletal health and overall health at every level.
- Tissue Injury, Repair and Replacement: It is well known that injuries to bones, joints, muscles, teeth and skin are extremely common and recover slowly, if at all. While many of these tissues are injured at a gross level, even more are likely to be damaged at a microscopic level. The accompanying pain and financial burden associated with injuries to these tissues can have a considerable negative impact on quality of life for individuals as well as their families. In light of this, there is an urgent need for highly innovative research into the cause and prevention of the physical, psychological, psychosocial and economic impact of these acute and chronic injuries. A more thorough understanding of the biological, clinical, psychological health services and population-based aspects of injury and repair is also required.
- Pain, Disability and Chronic Disease: Pain and disability (both physical and psychological) are very common sequellae of all chronic diseases (congenital and acquired) within IMHA's focus areas. In addition to relevant congenital conditions such as muscular dystrophy and osteogenesis imperfecta, a wide variety of acquired conditions such as arthritis, metabolic bone disorders, fractures, osteoporosis, periodontal disease and soft tissue injuries become more common with advancing age, for reasons requiring further exploration. Over the next two decades, the prevalence of many chronic diseases and conditions will increase, resulting in consumption of a far greater proportion of Canada's health care resources. To shed light on the mysteries associated with pain, disability and chronic diseases, IMHA will support research relevant to this strategic theme.
1.3 Evaluation Objectives and Issues
The Common Performance Measurement and Evaluation Framework (henceforth the Common Framework) was developed through a highly consultative approach and was approved by all 13 Institutes. It recommended that each Institute conduct a mid-term (formative) evaluation of its activities, outputs and outcomes at the end of the first funding cycle in 2005. The goals of this evaluation are the following:
- to provide Institutes with feedback on their overall progress and effectiveness at a point in time when such feedback can best be used to provide guidance for strategic decision-making about the direction of the Institute; and
- to provide input into the Five Year (Quinquennial) Review of Institutes required by The CIHR Act.
The issues addressed in this evaluation meet the needs of CIHR and Treasury Board requirements for formative evaluations. They are as follows:
- Relevance: To what extent is there still a need for this Institute to support the development of Canadian capacity and research excellence in this field of health research?
- Delivery: What has been the influence of other factors on the overall effectiveness of Institutes?
- Effectiveness: How effectively has this Institute achieved its objectives, fulfilled its mandate and mission, and achieved its vision? How effectively and uniquely has this Institute contributed to the overall objective of the CIHR?
- Alternatives: Are there alternative ways to achieve the same or better results in terms of research capacity, excellence and impacts in this research domain with greater efficiency?
The scope, issues, questions and methodology were approved by each Institute, by the Evaluation Steering Committee and by the CIHR Standing Committee on Performance Measurement, Evaluation and Audit.
The evaluation consisted of four main lines of evidence:
- a review of documents and administrative data relating to the IMHA;
- 43 key informant interviews with IMHA staff and IAB members, researchers and students, stakeholders and partners;
- case studies of two IMHA initiatives; and
- a telephone survey of 123 funded and 43 non-funded researchers affiliated with IMHA.1
2. Evaluation Results
2.1 Relevance
The Institute continues to be relevant. There is almost unanimous agreement across interviewees that IMHA's mandate and strategic priorities are appropriate, lending support to the relevance of the Institute. Also, because musculoskeletal diseases account for 10% of the economic burden of illness in Canada, second only to cardiovascular disease, there continues to be a demonstrable need for IMHA.
Though many interviewees feel that the research domains upon which IMHA focuses are all appropriate and fit together, a significant proportion, particularly among stakeholders, have concerns that skin, oral health and rehabilitation do not fit as well within the breadth of the mandate. Furthermore, representatives of these three research communities do not necessarily feel at home within IMHA. This may be because these three communities are not as well organized and visible and because IMHA focused somewhat more attention on the arthritis community early on. According to IMHA staff, this was a strategic decision so that IMHA could demonstrate results early on. The Institute's intention is to increase focus on less developed communities in the coming years.
Interview results indicate that most see the Institute as the most appropriate mechanism to make a difference in developing Canadian capacity, research excellence and knowledge translation in IMHA's domains. Many reasons were given for the appropriateness of IMHA, referring to a variety of aspects such as its focus on specific communities, the IAB and the Institute's connection to policy. Nearly all of funded researchers surveyed also agree that the Institute is needed. Although few specific alternatives to the CIHR model were suggested with little rationale to support them, a number of suggestions were made with regards to improving the existing mechanism related to such issues as funding, increased power/autonomy, and partnerships to facilitate knowledge translation.
2.2 Effectiveness
Interviewees and funded researchers surveyed generally agree that IMHA has made a contribution to the creation of new knowledge, although some interviewees feel that it is too soon to fully assess the contribution. Indeed, at this point there are certainly no quantitative measures of knowledge creation outcomes. A specific example of how knowledge has been created is the "tool box" designed to nurture creativity and therefore new knowledge. This tool box consists of a number of components, such as grants and awards, training programs and networks.
Views on the Institute's contribution to knowledge translation were considerably more mixed. While most staff and IAB members as well as researchers who felt qualified to respond do think that knowledge translation is a priority for IMHA, stakeholders disagree, reporting a need for knowledge translation strategies to better connect with research communities, partners and stakeholders. Just over half of interviewees feel that IMHA has contributed to knowledge translation, and many noted challenges such as a lack of understanding around knowledge translation and certain types or areas of research being more difficult to translate than others. Still, most funded researchers surveyed feel that IMHA activities support knowledge translation and the Institute Performance Report notes efforts to determine targets, current activities and gaps and opportunities, as well as activities including Requests for Applications (RFAs) that enable the best dissemination and updating methodologies, the incorporation of knowledge translation in all research activities, and a Knowledge Exchange Task Force.
While it appears from the data that interviewees and surveyed funded researchers agree that IMHA has successfully built research capacity, there were some concerns that not enough work has been done in the less developed research areas (skin, oral health, rehabilitation). Nevertheless, IMHA has sought to increase capacity primarily through training centres, and also through priority announcements to fund researchers whose applications scored highly but were not funded through CIHR's regular competitions. Most interviewees do feel that the Institute has increased the number of trainees it supports. Staff and IAB members noted a key challenge to capacity building: the success rate on grant applications. Increasing capacity without increasing available research funding will serve to diminish success rates that are already viewed by most as low.
The IMHA Performance Report cites a number of activities undertaken to provide opportunities for capacity development, such as needs analysis/environmental scan, research career programs and student exchange programs with other countries. The report also indicates that the number of trainees has increased.
In addition to human capacity, there is evidence that the Institute has also carried out work to develop infrastructure capacity, including the Invention, Tools and Techniques in Health Research initiative described in the Performance Report. Funded researchers surveyed also tend to feel that IMHA has contributed to capacity development in the research environment to at least some extent.
Interviewees who were able to comment generally feel that the Institute is contributing to CIHR's transformative vision, citing examples such as strategic programs that emphasize collaborative research and interdisciplinary activities. Funding data indicate that most funding related to the IMHA mandate has been in Theme 1 (biomedical research) and, to a lesser extent, Theme 2 (clinical research) though there has been a small increase in funding over the past five years for Theme 4 (research in social, cultural, environmental and population health). Still, a number of interviewees, particularly researchers, do not think that IMHA has contributed significantly in this regard.
Most interview respondent groups were not sufficiently familiar with the CIHR ethics mandate to comment on how well IMHA has contributed to it. IAB members who were able to comment described significant contributions, including the presence of an IAB member on the ethics board (although this position has had some turnover) and the inclusion of ethics at all IAB meetings. In addition, significant IMHA funding has gone to grants in ethics.
Interviewees and survey respondents agree that IMHA has made at least some progress in terms of achieving its mandate and strategic priorities, although it is too soon to truly assess the impacts. Still, administrative data indicate that IMHA is positively influencing research under its mandate; relevant expenditures are increasing.
Partnerships and collaborations are being developed; the Performance Report indicates 20 partnerships with organizations and other Institutes. Interviewees are generally satisfied with partnership activities, although some would like to have seen more. Greater partnering success was seen in the area of arthritis research; less success was had in other areas. Key barriers to partnering included insufficient human and financial resources or time and competing interests among organizations. Nevertheless, IMHA is seen to be collaborating well with other Institutes.
One method used by Institutes to influence research direction is the creation of funding opportunities in the areas of strategic and emerging research importance relevant to the Institute's research domains. The extent to which these opportunities generate responses from the research community is an indicator of the level of activity and interest in these research areas. Interviewees generally agree that IMHA has successfully influenced the research agendas within its relevant research communities. Findings from the Survey of Funded Researchers also support this conclusion.
2.3 Delivery
The Institute Performance Report and interviews with staff and IAB members indicated that IMHA employs an iterative approach to developing strategic priorities. The IAB assisted the Institute with identifying a large number of potential priorities and then narrowing them down to a more manageable number. Interviews with all key informants revealed four primary planning mechanisms that were effectively used by IMHA: a stakeholder survey, a consensus conference, the IAB itself and workshops. Most staff and IAB members are satisfied with these mechanisms, but stakeholders and researchers had mixed opinions. For these latter groups of respondents, there is a desire for more involvement and even a lack of knowledge of the planning process, particularly among researchers.
Interviews and results from the surveys of funded and non-funded researchers indicate that IMHA has been able to identify areas of emerging and strategic importance.
Interviews with staff and IAB members show a high level of satisfaction with the functioning of the IAB, although a few concerns were raised regarding turnover and the subsequent integration of new members. Some suggestions for improvement were also made.
Input and feedback are obtained from stakeholders primarily through the activities of the Scientific Director and IAB members, as well as through consensus conferences and surveys. IAB members are seen as a vital link to the research and stakeholder communities, keeping their relevant constituencies informed of IMHA activities and holding meetings across Canada.
The 2002-03 Annual Report indicates that IMHA has developed a comprehensive communications plan that incorporates many activities, including reorganization of the Web site, newsletters, a video and so on. Some interviewees, however, believe that improvement is needed, referring particularly to the challenge of identifying the IMHA research community. This creates difficulties in responding to the needs of the research community. Another issue with communications pertains to public awareness. There are concerns among stakeholders that the public has little awareness of IMHA.
There are some issues with the role of CIHR corporate. Most IAB members did not know any details of the operational relationship between corporate CIHR and IMHA, and those that did, as well as Institute staff, were not satisfied with the relationship. Concerns included a lack of responsiveness and coordination regarding communication and planning, insufficient resources and tensions between the two parties.
There is considerable agreement among interviewees that funding is not sufficient; it was suggested that disease burden and broadness of mandate be considered when determining funding levels for the Institutes. In particular, insufficient funding can be very discouraging for researchers when so many projects are not being funded. Data provided by CIHR corporate generally support this concern. Opinions were mixed among respondents about the current balance between investigator-initiated research and strategic research funding. Staff/IAB members and researchers were more likely to think the balance was appropriate, while stakeholders tended to prefer that more funding go to strategic research.
The evaluation found some evidence of dissatisfaction with the inclusion of social science-based research within the CIHR mandate. This dissatisfaction stems from a misunderstanding over the amount of funding going towards Themes 1 and 2 within CIHR relative to the funding available previously through the Medical Research Council (MRC). Some researchers interviewed feel that the CIHR focus on the four research themes has resulted in less funding to biomedical and clinical research. Funding data for CIHR and MRC indicate, however, that this is not true. In fact, total funding available to biomedical and clinical research has increased relative to what was available under the MRC.
3. Recommendations
Following are recommendations that emerge from this mid-term evaluation of IMHA. Please note that the recommendations appearing here are those that are Institute specific. Other recommendations will be made to appropriate bodies at CIHR corporate that are outside the span of Institute control.
Recommendation 1:
The Institute has been found to be doing well and is encouraged to continue the following:
- sustain its efforts in the areas of research excellence, capacity development and funding strategic priorities;
- maintain the current structure and operation of the IAB;
- sustain its activities in the area of partnering and collaboration;
- maintain its knowledge creation strategies using its "tool box" approach; and
- sustain its productive efforts in the ethics area.
The following areas are ones in which it is recommended the Institute take action to improve:
Recommendation 2:
Consultation and Communication - The Institute should work- with the support of CIHR corporate - to identify mechanisms to raise its profile with the public and to better communicate its successes. The Institute should also endeavour to identify and communicate with researchers whose research may fit within the mandate of the Institute but who do not currently identify with IMHA.
Recommendation 3:
Capacity Building - While the Institute is encouraged to sustain its overall substantial progress in the area of capacity building, the Institute may wish to consider increasing efforts on building capacity in the oral health, skin and rehabilitation research communities with the aim of better integrating all six research areas included within IMHA.
Recommendation 4:
Planning Mechanisms - While encouraged overall to maintain planning mechanisms, as they are effective, the Institute should consider identifying mechanisms for improving stakeholder engagement in the Institute's planning process.
Recommendation 5:
Knowledge Translation - The IMHA should review its efforts in support of KT. The efforts expended by the IMHA in the area of KT do not always appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IMHA are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT.
Recommendation 6:
Contribution to Transformative Vision - While encouraged to sustain its efforts in contributing to the transformative vision of CIHR through its inter-disciplinary work, the Institute should endeavour to better communicate the benefits of the transformative vision to researchers.
Recommendation 7:
Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results.
4. Management Response
Overall comments on the report, including, if desired comment on Recommendation 1 that suggests continuing certain activities that are going well:
| Recommendation | Response | Action Plan |
| 1. Consultation and Communication - The Institute should work- with the support of CIHR corporate - to identify mechanisms to raise its profile with the public and to better communicate its successes. The Institute should also endeavour to identify and communicate with researchers whose research may fit within the mandate of the Institute but who do not currently identify with IMHA. |
Agree |
IMHA totally agrees that more work is required with corporate communications to raise the profile of IMHA in the public. In addition, IMHA will endeavor to: 1) Raise its profile by partnering with various stakeholder groups and with existing networks in communicating IMHA activities, outputs and outcomes to their respective target audiences. IMHA's overall capacity remains much smaller than it should be, given the burden of illness IMHA represents. Therefore communications with researchers whose research may fit within the mandate of the Institute but who do not currently identify with IMHA continues to be a priority for the Institute's capacity development strategy. 1) IMHA will continue to expand its research community through communications and programs (including RFAs) that will attract researchers whose research may fit within the mandate of the Institute but who do not currently identify with IMHA.2) IMHA will expand its database to include those researchers that identify IMHA as a secondary Institute as a first step towards trying to reach this group of researchers. |
| 2. Capacity Building - While the Institute is encouraged to sustain its overall substantial progress in the area of capacity building, the Institute may wish to consider increasing efforts on building capacity in the oral health, skin and rehabilitation research communities with the aim of better integrating all six research areas included within IMHA. | Agree |
IMHA recognizes the need to focus on its smaller communities (skin, oral health and MSK rehabilitation) with community-specific strategies in progress for skin and MSK rehabilitation and oral health to be developed over the next year. 1) A renewed SWOT analysis in each community (focusing on capacity, program needs and some realistic capacity targets for each community) will be carried out. |
| 3. Planning Mechanisms - While encouraged overall to maintain planning mechanisms, as they are effective, the Institute should consider identifying mechanisms for improving stakeholder engagement in the Institute's planning process. | Agree |
We are very grateful for feedback on this point and will endeavor to identify better mechanisms of stakeholder and partner engagement in our planning mechanisms going forward in all of our communities. We definitely recognize the need to focus staff efforts in the three smaller communities that we represent (skin, oral health and MSK rehabilitation) to engage stakeholders and partners more effectively in the next few years. Specifically, the staff and IAB will endeavor to: 1) Follow up on the skin consensus conference held in December 2004 with the development of specific programs, partnerships and stakeholder plans with our IAB liaisons in that area. |
| 4. Knowledge Translation - The IMHA should review its efforts in support of KT. The efforts expended by the IMHA in the area of KT do not always appear to have a large profile among IAB members, researchers and stakeholders. The area should be reviewed to determine to what extent the efforts are likely to result in the impacts that the IMHA are hoping to achieve, and how the partnerships and collaborations that have been developed could be effectively applied to KT. | Agree | As noted in the evaluation, IMHA acknowledges that this has been a weak spot in some ways, and a strength in some other ways. This is a function of the potential breadth of KT activities that is possible and our desire to have a strong KT strategy in place before launching anything. While discussing it a lot at the IAB, IMHA has not broadly communicated any KT strategy thus far and KT components of IMHA's strategic research toolbox have been modest. We agree that a "KT strategy" is needed. 1) The Knowledge Exchange Task Force has been an IMHA priority, but we consider that it is still at an early stage since of inception, therefore its communication to stakeholders and its impact, thus far, have been modest. Its full impact will be monitored and better understood in the coming year. 2) A model that embeds KT in IMHA activities has been designed and will provide the template for a comprehensive KT strategy. This strategy is currently in development with the IAB for discussion with stakeholders. 3) IMHA plans to create a "KT module" for all future IMHA RFAs, with targeted KT areas for each IMHA community (clinical uptake and impact, policy change and commercialization/innovation being the top three KT areas identified for IMHA thus far). 4) IMHA also recognizes that capacity building of KT expertise is required in order to achieve desired targets and impact. This will also be embedded in the KT strategy. One of our initial plans includes implementing KT training modules into all of IMHA's training centres. |
| 5. Contribution to Transformative Vision - While encouraged to sustain its efforts in contributing to the transformative vision of CIHR through its inter-disciplinary work, the Institute should endeavour to better communicate the benefits of the transformative vision to researchers. | Agree |
IMHA recognizes the importance of communicating the transformative vision of CIHR to researchers and will continue to contribute to this joint effort collaboratively with other institutes and CIHR corporate. IMHA plans on: 1) Working with corporate communication to better communicate the benefits of the transformative vision to researchers.2) Continue to hold stakeholders' forum across the country more actively focusing on communicating evidence of the impact of implementing the transformative vision. 3) Building capacity in pillars 2-3 and 4 should help improve the balance of funding across the four CIHR pillars within IMHA and further contribute to the transformative vision and communications of impact thereof. |
| 6. Performance Monitoring and Reporting - In order to ensure that the Institute is achieving the results it intends to achieve, it is recommended that performance be systematically monitored and reported and, where possible, effective performance targets be put in place to measure results. | Agree | IMHA recognizes that monitoring performance and reporting is important to ensure that its targets are met. IMHA has had a subcommittee of the IAB on performance evaluation almost since its inception. 1) As of 2006, IMHA is developing a strategic "targets" document that will set targets in: capacity by focus area, by research priority areas and by KT output area. 2) IMHA will continue monitoring and providing yearly and quarterly reporting through this, Strategic/ target document, Annual environmental scan document, Annual report toC, Annual report of the Institute, Annual operational plan, Annual performance document, Quarterly budget report to IAB, Quarterly operational report to corporate. 3) As mentioned above, a renewed SWOT analysis in each community (focusing on capacity, program needs and some realistic capacity targets for each community) will be carried out. |
1Note that the survey of researchers was a cross-Institute survey conducted by EKOS Research Associates.
Supplemental content (right column)
- Modified: