Interim Evaluative Study of the Interdisciplinary Health Research Team Program and the Community Alliance for Health Research Program

Other format

Extended Executive Summary

Evaluation and Analysis Branch
March, 2006

Table of contents

  1. Introduction
    1. Purpose of Report
    2. The CIHR IHRT and CAHR Programs
    3. Approach to Development of Evaluation
    4. Methodology
  2. Findings
    1. Capacity Building
      1. Use of Existing Relationships
      2. Characteristics and Extent of Interdisciplinarity and Collaboration
      3. Training Students
    2. Project Management
      1. The Role and Impact of the University System
      2. Role of Community Partners
    3. Knowledge Translation
      1. Understanding of Knowledge Translation
      2. Mechanisms of Knowledge Translation
      3. Publishing
  3. Key Conclusions and Lessons Learned

1. Introduction

i. Purpose Of Report

The following is an extended executive summary of the interim evaluative study of CIHR's Interdisciplinary Health Research Team (IHRT) and Community-based Alliance for Health Research (CAHR) Programs. A larger report with more detailed discussion on methodology and results is available upon request.

ii. The CIHR IHRT and CAHR Programs

IHRT and CAHR programs were considered CIHR's two "transition" programs because they included elements of CIHR's mission that were distinct from MRC, namely: utilization-oriented research with an inclusive, broad disciplinary base, responsive to important health needs in Canadian society and the evolution of the health system.

A total of $79,862,748 was scheduled to be allocated to the two programs over five years. In January 2001, a total of 11 IHRT grants and 19 CAHR grants were awarded (amounts awarded ranging from $0.6M to $3.2M) for up to five years of funding. These involved roughly 600 researchers, representing over 100 institutions and 242 community-based partners.

The IHRT program was intended to increase the interdisciplinarity and applicability of health research in Canada, by encouraging health researchers from multiple disciplines to work together on programmatic research that formed a single, integrated entity. The program was intended to go beyond the traditional single-researcher, single-laboratory model. To be eligible for funding, the teams were to consist of five of more researchers from two or more of the biomedical, clinical, health services, or population health research themes, housed in two or more institutions, and working on an important health issue.

The CAHR program was intended to increase the responsiveness of Canadian health research to community needs; and to enhance mutual learning and collaboration among community organizations. Building on the Community-University Research Alliances (CURA) program operated by Social Sciences and Humanities Research Council of Canada (SSHRC), the program supported partnerships between researchers and communities, with the expectation that community partners would be full participants in all aspects of the research endeavour.

iii. Approach To Development Of Evaluation

The evaluation was formative, and similar in approach and scope to process studies as elaborated by Cronbach1, in which an evaluation seeks to understand and describe what occurs in the day-to-day practice of running and operating a program or activity, in this case conducting research funded under the IHRT or CAHR grants. The evaluation does not, however, seek to address the impact or long term effects of interdisciplinary/community-based research and does not provide management recommendations for the continuation or discontinuation of the program.

An Evaluation Steering Committee was engaged throughout the design phase and the first part of the evaluation, which was conducted by an external consultant, Applied Research Consultants (ARC). The committee consisted of 8 representatives of CIHR, the IHRT and CAHR programs, FRSQ and academia.

The IHRT and CAHR programs each have unique program elements and were regarded as distinct, though complimentary programs during implementation. While the focus of the evaluation is on general issues related to the conduct of large, diverse, team-based grants, the evaluation does highlight unique program findings where pertinent (e.g., the role of community-based partners for CAHR grants). The report provides data on the following three areas: Capacity building; Project Management; and, Knowledge translation.

iv. Methodology

Two separate methods were used in this evaluation: a focus group with IHRT and CAHR Principal Investigators and selected community-based representatives, and a survey of the entire IHRT and CAHR populations, including community partners.

  • Four separate focus groups were conducted during the summer of 2003 with 20 IHRT and CAHR representatives.
  • The survey was administered to an updated list of program participants. Fifty percent (50%) of the individuals contacted responded to the survey (n=112). Response rates were similar for the two programs, 48% for CAHR and 53% for IHRT.

The majority of focus groups, interviews and survey respondents were researchers, community-based participants and PIs directly engaged in IHRT and CAHR. Thus, there is a potential bias in the weight of evidence to support the IHRT and CAHR approaches, as well as the current management styles of the PIs and researchers. Without a valid comparison group, it is not possible to determine how much interdisciplinary or community-based research may have occurred in the absence of CIHR funding. The evaluation is, by necessity, exploratory given the lack of a standard definition of "interdisciplinarity" and the lack of common measures to assess its use and success.

2. Findings

i. Capacity Building

This section highlights interim evaluation findings related to the research capacity-building function of the IHRT and CAHR teams. In the context of the evaluation, capacity building referred to the manner in which large, team-based grants fostered and promoted interdisciplinary and collaborative research. In this section, findings are presented on:

a. Use of existing relationships

The focus group discussants indicated that members of most teams had some previous relationship or collaboration. Researchers describe the existence of a pre-existing 'informal collaboration that was "cemented" by the CIHR funding. The survey also established that almost three-quarters of researchers work in close collaboration with researchers from other disciplines and about half of those had previously collaborated with some of those researchers on previous grants.
Without a valid comparison group, there is no way to ascertain whether interdisciplinary and community-based research teams need to have had previous relationships. However, it appears that most grant teams used pre-existing relationships in order to submit a successful proposal.

b. Characteristics and extent of interdisciplinarity and collaboration

PIs who participated in the focus groups observed that CIHR did not define interdisciplinary research and had offered little guidance on its expectations for interdisciplinarity other than the structural requirements of the proposal. The call for proposal defined the four CIHR research themes and required IHRT grants to span at least two research themes. However CAHR grants were only required to involved community partners: the call for CAHRs did not explicitly require members from different research themes.
The survey did provide preliminary quantifiable characteristics of interdisciplinarity and collaboration as it occurred within the IHRT and CAHR programs. Respondents identified those characteristics that applied to their grant and if they selected more than one, they ranked the relative importance of each. Table A-1 shows the details, listing the principal grounds for the interdisciplinary nature of the research grant in descending order of importance.

Exhibit A-1 Average Ranking of Principal Grounds for Interdisciplinary Character of the Grant, by Grant Type

Grant Type
Total Cahr IHRT
Total (multiple responses allowed) (n=116) (n=67) (n=49)
Dependence of the research on complementary skills and/or knowledge 1.78 1.85 1.67
Personal relationships that existed prior to the grant among key participants 2.09 1.89 2.35
Commonality of research concepts among participants 2.58 2.44 2.76
Recognition of new integrative concepts among participants 2.59 2.34 2.97
Dependence of a number of the participants on sharing technology such as data banks or major items of equipment 2.90 3.08 2.78
Other 3.17 3.17 0

Base: Principal Investigators, Co-Applicants, Lead Investigators and Representatives of Community Partners.
Question: From your perspective, what are the principal grounds for the interdisciplinary quality or character of this research grant? Please identify all that apply. If you choose more than one, please rank them with 1 indicating the most significant.

The order of the interdisciplinary characteristics is identical for CAHR and IHRT grants with the most important characteristic of interdisciplinarity being the dependence of the research on complementary skills and/or knowledge. The second most important reason was the existence of personal relationships prior to the grant among key participants. The results of table A-1 appear to suggest that interdisciplinary research is primarily based on interdependence between researchers, be it complementary skills or established relationships between researchers. What is not known at this point is how the above factors actually influence the development of research issues and the on-going management of large research teams.

The survey also asked researchers to identify the extent to which the grant involves them in interdisciplinary work, using a set of categories that emerged from the Kessel et al.2 analysis and the focus group discussions. Table A-2 provides four descriptions of work on a grant, arranged to reflect increasing levels of interdisciplinary work. The descriptions are not mutually exclusive and the survey allowed multiple responses. The data show similar distributions for CAHR and IHRT participants and suggest that the majority of both IHRT and CAHR researchers are involved in the design of research that actively involves other disciplines over a substantial period of time, which represents the highest level of interdisciplinary research. Though not shown in table A-2, the most junior members of the team and trainees had the lowest scores on the last category, suggesting that senior researchers or PIs tend to be those more actively engaged with researchers from other disciplines.

Table A-2 Extent of Interdisciplinary Work by Grant Type

Grant Type
Total Cahr IHRT
Total (multiple responses allowed) (n=141) (n=76) (n=65)
Work almost exclusively within my discipline 11% 9% 12%
Work within own discipline in parallel or sequential collaboration, with little blending of disciplines 32% 28% 37%
Import citations, instruments or techniques from other disciplines 37% 41% 32%
Contribute to the design and execution of research that actively involves other disciplines over a substantial period of time at each stage of the research 63% 62% 65%

Base: Principal Investigators, Co-Applicants, Lead Investigators, Researchers, Post Doctoral Fellows and Trainees.
Question: We are interested in the extent that the CIHR grant involves you in interdisciplinary work. Please indicate if these descriptions apply to your work on the grant.

As an exploratory methodology, the evaluation team also conducted a factor analysis of the survey in order to develop a quantitative index of interdisciplinarity. This index was based on the extent to which researchers perceived themselves to be working in interdisciplinary ways. The characteristics of interdisciplinarity, based on self-reported survey items, included role on the project, number of collaborators involved in the project, the extent to which the respondent worked within their own discipline and the extent to which the respondent contributed to the design and execution of research that actively involved other disciplines3. Using the index, the evaluation team then attempted to determine what characteristics are associated with working in an interdisciplinary way. The analysis is exploratory and the results should be treated with some caution. Preliminary findings, however, suggest:

Interdisciplinarity is associated with:

  • Role in the grant. Principal Investigators were more likely to indicate they worked in an interdisciplinary way than researchers hired to work on the grant.
  • Appointment at the university. Pair-wise comparison suggested that research associates and post-doctoral appointments engaged in less interdisciplinary work than researchers who held part and full time tenure-track and full time non-tenure track appointments.
  • Co-authored paper with a community partner is associated with more extensive interdisciplinary work.
  • Effect of differences in publication record on academic career. Respondents who reported higher levels of interdisciplinary activity also identified more favourable effects of their publication record on their academic careers.

Interdisciplinarity is not associated with:

  • Program. Researchers in CAHR and IHRT grants described similar levels of interdisciplinarity in their work.
  • Status at the university. No difference in interdisciplinary activity according to status (Professor, Associate Professor, etc.)
  • CIHR Pillar. No significant differences across pillars.
  • Number of publications submitted first or senior author.
  • Recognition of contributions such as professional and community service.

c. Training Students

It appears that grants in both programs are making extensive efforts to train students. During the focus group, CAHR researchers were particularly likely to report efforts to develop the research competence of students and of staff members from community partner agencies. The survey respondents include only a small number of Master's students, doctoral candidates and post doctoral researchers. Their responses seem to indicate satisfaction with participation in the grant.

The survey asked about the impact of the grant on the structure of graduate programs at the university. Table A-3 shows that over half of the researchers involved in the two grant programs report either new interdisciplinary programs had been developed or increased flexibility had been introduced into existing programs.

Table A-3 Influence on Graduate Programs by Type of Grant

Grant Type
Total Cahr IHRT
(n=141) (n=76) (n=65)
New interdisciplinary programs developed 23% 21% 26%
Increased flexibility in existing programs 35% 26% 42%
Other (Specify) (most respondents who selected this category indicated no influence/impact had occurred, none was require, etc.) 40% 42% 38%

Base: Principal Investigators, Co-Applicants, Lead Investigators, Researchers, and Post Doctoral Fellows and Trainees.
Question: How has the grant influenced the structure of graduate programs?

While the data reported in table A-3 is tentative due to the possibility of double-counting, the data do indicate that interdisciplinary and/or community-based grants may lead to University-based changes such as the creation of new courses and opportunities for training. No follow-up has been conducted at this point to obtain more information on the precise nature and examples of these changes.

ii. Project Management

The interim evaluation findings highlighted here are those areas that may be instructive to understanding the opportunities and challenges related to running interdisciplinary and community-based research grants, such as:

a. The role and impact of the University system

The focus group respondents, particularly the academics, expressed concern that interdisciplinary research may not yet be widely accepted within Universities and expressed some frustration that the level of work required to manage an IHRT or CAHR was not necessarily widely acknowledged within Universities. Survey responses tended to confirm the focus group discussion. As table B-1 indicates, only about one quarter of the 22 Principal Investigators who completed the survey indicated that their IHRT or CAHR grant addressed the department's highest priorities, though approximately two thirds indicated the grant fell within departmental priorities. Further exploration would be needed, however, to understand how large, team-based grants like IHRTs or CAHRs are perceived with University departments. The lack of recognition may be a function of the topic areas rather than the interdisciplinary nature of the grant.

Table B-1 Grant Related to PI's Departmental Priorities by Program

Grant Type
Total Cahr IHRT
(n=22) (n=13) (n=9)
The grant is only marginally related to departmental priorities 5% 8% 0%
The grant falls within departmental priorities 64% 62% 67%
The grant addresses the department's highest priority 27% 23% 33%
Comment 5% 8% 0%

Base: PIs.
Question: How well does the grant fit with the priorities of the departments to which the PIs are appointed?

An additional concern, expressed during the focus group, was that junior faculty faced multiple disincentives to participate in the grant:

A major aspect is the impact of the work, e.g., changing clinical practice. If junior people are involved in important work, they should get credit even if the work is interdisciplinary and they are one among a number of contributors. People must forego opportunities for grants that lead to publications in order to participate in (CAHR/IHRT) grants that are not monetarily rewarding and not career advancing." Universities encourage junior faculty to get first author publications and grants where they are the PI. An Assistant Professor will not likely have the breadth of experience to pull together a grant such as this. And they don't get credit for contributing to the CAHR/IHRT grant from tenure review committees. The grant is time consuming and they won't get publications quickly. Further, most tenure review committees are within disciplines. They have a great deal of difficulty understanding if a researcher is making a significant contribution.

At this point, it is not possible to ascertain what, if any, long term impact participation in these large team grants has on academic performance and promotion. One possible strategy for future studies may be to track the career progression of young faculty who started in IHRT or CAHR grants and compare their progress, including publications and promotions, to young faculty that started with more "traditional" grants (e.g., single lab, independent research grant).

The focus group discussants, including a cross-section of both IHRT and CAHR recipients, also indicated that the research ethics review process had created significant problems for several grants:

Multiple Research Ethics Board (REB) approvals—one grant working with 104 hospitals requires 104 approvals.
University ethics committees are rarely knowledgeable about the ethics related to community-based research, particularly for high-risk populations.
There are no guidelines for these reviews, so committees tend to deny. If there is any ambiguity, don't do the research."
There is a huge lobby by ethicists whose goal is to make the review complex, a scientific discipline. The researcher's goal is to provide knowledge about populations. Professional ethicists look for how to stop 'exploitative researchers' from damaging vulnerable people.

However, it appeared that the research ethics approval process presented major challenges to a minority of grants. In part, difficulties may arise from what appears to be un-realistic expectations about the time and resources required to complete the process.

b. Role of Community Partners

In the focus group discussions, CAHR PIs addressed the nature and formality of relationships with community partners4. All are concerned about the stability of these central relationships. There seems general agreement that the relationship must go beyond close involvement with a single individual from a community partner, recognizing that when a key contact leaves, the relationship may not be sustained. At least two approaches were identified:

  • Establish more formal relationship. This approach suggests the partner and the grant sign a letter of understanding to formalize the relationship. Some PIs felt that such formal arrangements may not be worth the effort, however, since they are not binding. One PI commented: "A formal memorandum of understanding with an agency was not helpful when the agency disappeared. However, the commitment of the individuals continued."
  • Close involvement with agencies. A number of PIs described the importance of fostering relationships with the community and getting community buy-in. Some build community partners into the governing structure with representatives of partners appointed to governing bodies and/or advisory committees, giving them a formal role in planning the research. The reciprocal arrangements are also important, with researchers serving as board members of partner agencies.

According to IHRT PIs, about one-third of IHRT grants involved community partners. However of 44 community partners who responded to the survey, only two indicated they participated in an IHRT grant. The low response rate for IHRT community partners could reflect a variety of factors, failure of the PI to identify the community partner in the list of participants sent to CIHR, partners may not have consented to participate in the survey or they may not have responded to the invitation to complete the survey. Because almost all community partners who responded to the survey are associated with a CAHR grant, the following comments on the views of community partners should be interpreted to reflect the views of CAHR partners.

PIs, co-applicants and representatives of community partners described the nature of the links between community partners and the research grants. All CAHR respondents and those IHRT respondents who indicated their grants involve community partners, responded to this question. The data are shown in Table B-2.

Table B-2 Involvement of Community Partner Representatives by Program

Grant Type
Nature of Links with Community Total Cahr IHRT
Partners (multiple responses allowed) (n=111) (n=90) (n=21)
Community partner representatives are involved in the strategic decisions for the focus of the research activities 60% 63% 43%
Community partner representatives are actively involved in the grant, e.g., appointment to advisory or management group for the research project, appointment to the board / advisory group 58% 61% 43%
There is an informal agreement about the roles and responsibilities of the researchers and the community partner(s) 47% 49% 38%
We have signed a formal letter of understanding between the grant and community partner(s) 37% 38% 33%
Community partner representatives are involved in the strategic decisions for the research grant, e.g., allocation of resources for the grant 35% 39% 19%
Researchers are actively involved in the partner organization, e.g., appointment to the board / advisory group 26% 27% 24%

Base: PIs, Co-Applicants and Community Partners.
Question: Links between community partners and the research grant can take many forms. Please describe the nature of the links /with our organization/ in your project.

While there is varying degrees of community involvement in both of these programs, the data confirms the overall close involvement of community agencies as described by focus group participants. Over half of all respondents indicated that community partners are involved in strategic decision on the focus and research activities and actively involved in an advisory or management group. About one-third of respondents indicated that community partners were involved in strategic decisions involving allocation of resources for the research grant.

In the focus groups, PIs described collateral benefits for researchers and agencies arising from these close relationships. Each lends legitimacy to the other's interactions with the community, such that a community partner's work gained stature from the association with university-based researchers and the researchers' work was validated in the eyes of the community.

Fifty-seven percent of community partners who responded to the survey indicated that involvement with the grant had changed their organization. A review of the description of those changes suggests they fall into two general categories:

  • Evidence-based decisions. About one-third of community partner respondents mentioned evidence-based decision-making, sometimes described as changes in services in response to research results.
  • Enhancing the research culture. About the same proportion of community partner respondents described an enhancement in the research culture within the partner organization (strengthen links with university-based researchers, new projects identified, increased capacity to participate in research and interpretation and dissemination of research results).

Community partners stressed the time and trust required to build effective partnerships, that the process is complex, requires open communication, careful listening on both sides, respect, and that researchers may have to deal with differences in the language (e.g. terminology) used by researchers and partners. The process cannot be rushed and the relationship can be difficult to maintain over time.

iii. Knowledge Translation

While knowledge translation is clearly a priority for CIHR, in much of the scientific community this emphasis is new.  The CIHR knowledge translation framework, in fact, was not articulated until about two years into the lifespan of these grants. In the following section, we examine the preliminary efforts made by IHRT and CAHR teams to implement knowledge translation strategies. The section is divided into three sections: understanding of knowledge translation, mechanisms of knowledge translation and publishing.

a. Understanding of knowledge translation

During the focus groups, CAHR PIs had different views of knowledge translation when compared to IHRT PIs. The principal concerns among CAHR PIs were "getting the results out there" so they can be understood, assimilated and applied. Concerns revolve around two issues:

  • Availability of appropriate vehicles for publication. A number of CAHR PIs commented on the difficulty of finding journals that will accept articles on community-based multidisciplinary research such as would be submitted by a CAHR team. One grant is considering one or a series of books to share the lessons learned in doing the grant and the findings of the research.
  • The resources required for knowledge translation. A number of PIs commented that they had underestimated the complexity of the task and the resources required.

We have not set up adequate resources for dissemination of research findings or to evaluate the dissemination. The success of the research has created a series of need and increased demand. We are victims of our own success.

IHRT PIs were much less clear about knowledge translation and what they were expected to do. They expressed frustration with the lack of guidance on CIHR's expectations of the grants. Knowledge Translation was a new concept to some, and not at all clear.

We need guidance from CIHR, including a concrete definition of knowledge translation and a better idea of how to do it

b. Mechanisms of Knowledge Translation

Respondents to the survey indicated that almost all grants take advantage of the vehicles traditionally used by academics, presentations at conferences and publications in peer-reviewed journals, to communicate their results. However a majority of grants also use the other vehicles named in the survey, websites, newsletters and the media. Some respondents commented that the list was very limited; that it did not include many of the mechanisms used by their grant to communicate research results to policy makers or managers who may use those results to inform policy positions or management decisions.

c. Publishing

While the focus group discussions indicated that IHRT and CAHR researchers were experiencing difficulty in publishing their results, the survey results suggested less reason for pessimism. About half of all respondents (54%) saw no difference in their rate of publication as sole or first/senior author since the beginning of the grant. CAHR researchers were less likely than IHRT researchers to report difficulty finding journals that will consider and accept interdisciplinary articles and articles in their discipline (68% CAHR and 85% IHRT). The survey enquired if researchers had co-authored a paper with a community partner. It is noteworthy that about an equal proportion of CAHR and IHRT researchers (28%) have done so. The survey also showed that many researchers have submitted at least one publication involving other disciplines. The proportions of CAHR and IHRT researchers were almost identical (74%).

3. Key Conclusions And Lessons Learned

  1. The Canadian Institutes of Health Research should ensure that program objectives and the measures by which program recipients are judged are clear and concise; the lack of direction from CIHR regarding key concepts such as interdisciplinarity and the knowledge translation goals of the grants appeared to cause frustration and anxiety in the research community during the period of this study.
  2. While the evaluation was limited in the extent to which it could rely on precise, quantitative and agreed-upon measures of interdisciplinarity, the data gathered do point to factors that are associated with working in an interdisciplinary environment. These measures could form the basis for on-going monitoring of further interdisciplinary funding.
  3. Evidence presented suggest that the IHRT and CAHR grants, in spite of their differing program mandates, definitions and operating restrictions, exhibited remarkably similar traits. For example, though there was no specific requirement that CAHR programs contain multiple disciplines within the grant, a large number nevertheless indicated a high level of collaboration between team members from different disciplines. The findings suggest that large, long-term grants with a focus on complex health issues may exhibit similar characteristics in the long-run.
  4. The results presented in the evaluation suggest that interdisciplinary research is based on interdependence between researchers, be it complementary skills or established relationships between researchers. The findings suggest that existing relationships prior to the grant may facilitate on-going interdisciplinary research.
  5. Early evidence indicate positive results from the integration of community partners into the research process, including beneficial results for the community-based partners themselves (e.g., more evidence-based decision-making). Examples were also presented regarding the mechanisms used by research teams to integrate community-based partners. There was disagreement, however, regarding the extent to which there should be formal linkages between the researchers and community-based partners, suggesting that the most effective mechanism or mechanisms for the integration of community-based partners has not yet been established. The issue is clearly complex and requires careful further analysis.
  6. Quantitative and qualitative evidence suggest that there are minimal incentives and opportunities for junior faculty to participate in interdisciplinary and/or community-based research. The length of time required to conduct research before publishable findings are available appears to be one major issue, though there appear to be other factors as well. Future studies may want to examine the effectiveness of large, team-based grants as vehicles for building research capacity amongst junior faculty.
  7. There may be a limit to the number of institutions that can and should be involved in a single grant, managed by single Principal Investigator. The evidence indicating that some grants were delayed due to multiple ethics board processes, for example, suggest that there are practical restrictions to the number of institutions that can be effectively integrated into a grant with a limited time period.
  1. See, for example, Cronbach (1982) Designing evaluations of educational and social programs
  2. Kessel, Frank; Rosenfield, Patricia & Anderson, Norman (2003), Expanding the Boundaries of Health and Social Sciences: Case Studies in Interdisciplinary Innovation
  3. Appendix A of the larger evaluation report contains more details on the index methodology.
  4. The eligibility criteria for CAHR community partners are based on SSHRC's guidelines for the Community-based Research Alliance (CURA) program. CAHR community partners must have been non-profit organizations located in Canada with a research mandate and conform to protocols of research that include freedom in the conduct of research and the obligation to publish findings

Supplemental content (right column)