Summative Evaluation of the Regional Partnerships Program (RPP) - Final Report: Part 2

Return to Table of Contents

Table of Contents

Has RPP Been Effective?
Per Capita Funding Gap Analysis
Per Capita Expenditures
Provincial Wealth
Competition Success
Conversion Rate

Has RPP Been Effective?

Two approaches have been used to address this question. One is the analysis of CIHR data bases to examine whether there is a funding gap - a comparison of (a) the actual funding earned within a province with (b) what would have been received within the province if CIHR funds had been distributed strictly on the basis of the population of the province. The other was an analysis of the later success in CIHR competition of those who had received RPP funding.

Per Capita Funding Gap Analysis

A central rationale for the establishment of RPP was the argument that four provinces with medical schools (NL, PEI, MN, SK) were obtaining MRC funding at a lower level than would have been expected on the basis of the province's population relative to the population of Canada. The present evaluation pursued this argument in a number of ways including a "gap" analysis in all six of the provinces with RPP allotments. For this analysis, two values were obtained for every year from 1981 to 2003 inclusive. These values are displayed for each of the RPP provinces in Figures 2 through 7.¹¹

Actual CIHR Funds	the funds earned from CIHR competitions in each year. The funds provided by RPP are included in these amounts but not partner funds. The inclusion of the RPP amounts has a progressively more important impact when the base to which it is added is lower.
Expectation	the CIHR funds that would have been received in a province had the funds been apportioned solely on the basis of the province's population as a percentage of the total population of the ten provinces.

A non-statistical interpretation of Figures 2 through 7 suggests the following:

NL	Although very close in 1981, throughout the following years the two lines diverge. Visually, the slope of the actuals line increases as of 1999. This may be viewed as an improvement even though the gap between the two lines continues to increase. There appears to be a levelling of the line in 2002 but it is too soon to know if that has continued to be the case. In 2003, the actual ($3,517,000) was 37% of the expectation ($9,447,000).
PEI	The two lines start at different points and, throughout the time period, slowly diverge. Visually, the slope increases for the actuals line as of 1999. This may be viewed as an improvement even though the gap between the two lines continues to increase. In 2003, the actual ($710,000) was 28% of the expectation ($2,505,000).
NS	From 1981 to 1991, the two lines are almost the same. The actuals then begin to drop below the expectations until, by 2003, the actual ($13,776,000) is 81% of the expectation ($17,019,000).
NB	The actuals line is consistently below the expectation line and the separation increases over time. In 2003, the actual ($709,000) was 5% of the expectation ($13,648,000).
MN	This is the one RPP province where the actuals line exceeds the expectations line from 1981 into the mid-1990's. The actual dips below expectation around 1998 and the relatively small gap widens slightly over time. In 2003, the actual ($16,923,000) was 80% of the expectation ($21,142,000).
SK	The two lines are quite close in 1981 after which they start to diverge and continue to diverge. In 2003, the actual ($8,649,000) was 48% of the expectation ($18,089,000).

There had not been established a criterion by which to determine how much of a gap was needed in order to conclude that the difference between actual and expectation is of a magnitude that requires special effort such as provided by RPP. If the criterion was 20%, then two of the six provinces (MN and NS) were just at the criterion threshold
in 2003.

As of 1999, the slope of the actual line takes a noticeable turn upward in all RPP provinces although this is rather slight for NB. Of course, a portion of this upturn is simply the addition of the RPP dollars to the other funding obtained within the province from CIHR competitions.¹² It would appear that the influx of RPP funding has had a positive effect upon the slope of the actual line. However, it should be recognised that this increase in actual, from 1999 forward, also coincides with the increased funding that became available with the establishment of CIHR and which also appears in the non-RPP provinces. This is apparent in the gap analysis data for non-RPP provinces included in Annex A.

Figure 2: NL Funds "Gap" from Population Expectation

Figure 3: PEI Funds "Gap" from Population Expectation

Figure 4: NS Funds "Gap" from Population Expectation

Figure 5: NB Funds "Gap" from Population Expectation

Figure 6: MN Funds "Gap" from Population Expectation

Figure 7: SK Funds "Gap" from Population Expectation

Per Capita Expenditures

Figure 9, below, displays the CIHR dollar expenditures per capita for all years beginning with 1981. It was calculated by taking the number of MRC/CIHR dollars for a given province and year and dividing by the population for that province and year as well as the totals for all provinces.

Table 1 is an interpretation from a portion of Figure 9 for the year 2003. It shows that, for that year, CIHR expenditures were the equivalent of $18.12 per person across all of the provinces. The per capita spending in each province is shown and the variance calculated by taking the difference between spending in a province and spending for Canada. The differences are very apparent.

Figure 9 is based on per-person CIHR expenditures. This is a different approach than was used for the gap analyses displayed in Figures 2 to 7. Although the two are different approaches to the data, both are in agreement with each other and reveal the same trends over time.

Table 1: Variance in Per Capita CIHR$ - 2003

2003	$/capita	Variance
Quebec	$23.12	+$  4.99
Alberta	$21.93	+$  3.81
Ontario	$18.71	+$  0.59
Canada	$18.12	$  0.00
Nova Scotia	$14.72	-$  3.41
Manitoba	$14.55	-$  3.57
British Columbia	$13.91	-$  4.21
Saskatchewan	$  8.69	-$  9.43
Newfoundland	$  6.77	-$11.35
Prince Edward Island	$  5.15	-$12.97
New Brunswick	$  0.94	-$17.18

Provincial Wealth

Provincial wealth is a very approximate index of the capability of a province to support health research. The relative wealth of all ten provinces is shown in the following histogram. As expected, the GDPs of the six provinces with RPP are relatively smaller than the other four provinces. There are also important differences within the six provinces hosting RPP. There are many who believe that the level of provincial wealth influences the infrastructure for health research and the numbers of health researchers. That hypothesis was not studied in this evaluation. However it is obvious that the four non-RPP provinces have significantly larger GDPs than the RPP provinces. It is also apparent that there is a close agreement on the rank order of the provinces whether for wealth, as measured by GDP, or by CIHR funding, as measured by actual funds received within the province.

Figure 8: Provincial GDP - 1999

Competition Success

It is generally accepted that a key indicator of success for the RPP is the performance of researchers in the open competitions of CIHR for operating grants following the receipt of support from RPP funding. To investigate the performance of "RPP-graduates", the records of RPP Principal Researchers in successive CIHR competitions were examined.

To enable this examination, the CIHR data bases were searched by CIHR personnel ¹³ and a list of all post-RPP competition activity was developed for each of the RPP Principal Investigators. These lists were then modified in the following ways:

• RPP awards prior to 1999 were removed.
• Projects not successful in CIHR competition and eligible for RPP (a second time) but not funded within a province's allotment were removed.
• Submissions to very recent competitions, for which outcomes have not yet been announced, were removed.

Figure 9: CIHR Expenditure Per Person

Note the sharp rise after 1999 when research funding increased. The dark thick line is the average for Canada. It can be seen that only QC is consistently above the Canadian average, AB is above average after 1988; NB, PEI, NL and SK are consistently below the Canadian average.

The review of the list for each Principal Investigator sought to identify the first instance of a successful outcome (as a PI or as a Co-applicant) in an operating grant application; that is, full funding ¹⁴ of the submission. That instance was then rated as described below.

Depending on the date of a RPP award, the opportunity window for full funding was from five years to six months. For each researcher who had received an RPP award, all post-RPP competitions were searched and a list was made of those which showed activity from that researcher as well as the outcome of that activity.

Three members of the evaluation team ¹⁵ assessed the activity record for each researcher and assessed the level of funding performance by addressing the following questions. Was the researcher successful in obtaining full funding and, if "Yes", was the success in an open competition or a strategic competition? If neither, was the researcher deemed eligible within his province for an RPP award and was such an award given? This last point required cross-checking against another data base to ensure that partner funding had been provided to support the RPP award.

The evaluation team also made judgments, based on the titles of the projects, of whether the post-RPP award was for the same substantive subject area as had been the RPP project. The judgment reached for each was one of converted (same subject area), not converted (different subject areas) or not clear (evaluators unable to decide). These judgments are subjective; given the use of the "not clear" category, there is reasonable confidence that the ratings are reliable.

Only the aggregates across the six provinces are reported in Tables 2, 3, 4 and 5.

Table 2: Success Rates Over Multiple Competitions

Category of Activity	All Post-RPP Activity
	Number	Per Cent
No post-RPP CIHR activity	36	24%
Open Competition Approved	80	54%
Strategic Competition Approved	26	18%
Funded for second RPP	6	4%
Totals	148	100%

The post-RPP success rate (obtaining non-RPP funding after having had RPP funding) for these 148 researchers is 72% (54% plus 18%). It is, unfortunately, not possible to know if this is an unusually high success rate, for the statistics span several competitions. A researcher could, theoretically, have entered up to 10 competitions in the period of time since obtaining an RPP grant or award (the 'opportunity window' for full funding post-RPP being from six months to five years) and succeeded only on the tenth try to be counted a success. We know of no such case but it must be considered as setting the context for understanding success rate over multiple competitions. From the point of view of the RPP objective, which includes supporting researchers seeking to be increasingly successful in CIHR competitions, a 72% success level is certainly encouraging.¹⁶

Conversion Rate

An examination of those submissions approved in open competition shows that at least 41% had converted; that is the researcher has gone on to obtain full funding for the continued research in the same substantive subject area studied in the RPP project. If one was to set aside the projects for which a clear decision on converted / not converted could be made, then 46% (33 of 71) would be considered to be conversions to full funding.

Table 3: Success Rate in Open Competitions

Open Competition Conversions	Open Competition Approved
	Number	Per Cent
Converted	33	41%
Not Converted	38	48%
Unclear	9	11%
Totals	80	100%

An examination of those submissions approved in strategic competition shows that at least 41% had converted; that is the researcher has gone on to obtain full funding for the continued research in the same substantive subject area studied in the RPP project. If one was to set aside the projects for which a clear decision on converted / not converted could be made, then 36% (8 of 22) would be considered to be conversions to full funding.

Table 4: Success Rate in Strategic Competitions

Strategic Competition Conversions	Strategic Competition Approved
	Number	Per Cent
Converted	8	31%
Not Converted	14	54%
Unclear	4	15%
Totals	26	100%

The rules of RPP allow the award of a second RPP under either of two circumstances: the first RPP was provided as a training award and/or there is at least a six month hiatus from the completion of one RPP and the start of a successive RPP. Of the six provinces, only two made use of this provision. Of the six such uses of a second RPP, five were in one province and the sixth was in another second province. A seventh person eligible for a second RPP did not receive RPP funding and he is not part of the group in the Table 5.

Table 5: Second RPP Projects

Second RPP Conversion	Funded for Second RPP
	Number	Per Cent
Converted	3	50%
Not Converted	2	33%
Unclear	1	17%
Totals	6	100%

---

¹¹ Dr E. Bruce Waygood is thanked for his suggestion to display these "gap" results in this reader-friendly fashion.
¹² Not included in these calculations are the matching funds which, for two provinces (NB and PEI) are significant additions.
¹³ The evaluation team is most grateful to the careful work conducted by Caroline Shewchuk for this analysis as well as for a number of other analyses dependent upon extractions from the CIHR databases.
¹⁴ The term "full funding", even though not exactly correct, will be used throughout this report to refer to those projects which are above the cut-off threshold established by the peer-rated competitions managed by CIHR. The grant or award may not be for the full amount requested but it will be for an amount approved through the review process and accepted by CIHR budgetary requirements. The full funded projects, of course, do not come to an RPP Advisory Committee to be considered for RPP funding.
¹⁵ Gavin Lemieux and Mathieu Cyr of CIHR and Gerald Halpern of Fair Findings.
¹⁶ Based on information from CIHR, this rate is certainly higher than the typical success rates of initial awards or the success rates for renewals in CIHR competitions.

[Table of Contents] 
[1] [2] [3] [4]
[Appendix A]
[Appendix B]