Abstract

Background: Modifiable risk factors for type 2 diabetes are primarily lifestyle related. However, we know little about the impact of low-intensity community-based lifestyle interventions on health and health services use. In this patient-oriented research, we sought to assess the effectiveness on quality of life of a 6-month, person-centred community-based lifestyle intervention (additional to usual care) for community-dwelling older adults (≥ 65 years) with diabetes and at least one other chronic condition, and their caregivers compared to usual care. Methods: We conducted a type II hybrid effectiveness-implementation randomized controlled trial (RCT) at two sites in each of three Canadian provinces. Knowledge user partners were engaged throughout. Participants were eligible if aged 65+ years, diagnosed with diabetes and multimorbidity, enrolled in a primary care setting or diabetes education program, capable of providing consent, and spoke English or French. Enrolled participants were randomly assigned to intervention and control arms (1:1). The intervention arm consisted of usual care plus: 1) up to 3 home/telephone visits; 2) up to 6 monthly group education sessions; 3) ongoing nurse-led care coordination and system navigation; 4) caregiver engagement/support; 5) monthly interdisciplinary team case conferences; and 6) collaboration with primary care, as needed. The control arm received usual care. The primary outcome was quality of life (measure: SF-12 Mental Component Summary [MCS]); intention-to-treat was used with missing data multiply imputed. Secondary outcomes and sensitivity analyses were tested. Results: The trial ran from July 2019 - May 2022; sites started at different points within this period. Of 619 eligible participants, 295 (48%) were enrolled and 246 (83%) completed 6-month data collection. Our primary analysis showed no difference between groups in the SF12-MCS (mean difference: ?0.71, 95% CI ?2.66 to 1.24, p = 0.47) or secondary outcomes (e.g., anxiety, self-care, physical activity); sensitivity analyses showed consistent results. Discussion: A low intensity 6-month community-based lifestyle intervention for community-dwelling older adults (≥65 years) with diabetes and at least one other chronic condition did not improve quality of life or other outcomes, in contrast to our previous studies. The trial ran during COVID-19, with disruptions and the shift to virtual delivery potentially diluting intervention effects.

Keywords

Diabetes, Older Adults, Multiple Chronic Conditions, Patient-Oriented Intervention, Pragmatic Effectiveness-Implementation Hybrid Type II Trial, Community-Based Intervention

Share and Cite:

1. Introduction

Diabetes is one of the most common chronic conditions worldwide, and its prevalence is rising (https://www.who.int/news-room/fact-sheets/detail/diabetes). Older adults (age ≥ 65 years) have the highest prevalence of diabetes of any age group (https://www.statcan.gc.ca/o1/en/plus/5103-diabetes-among-canadian-adults)

and multimorbidity is common with upwards of 40% of older adults with diabetes having three or more chronic conditions [1] [2]. Models of care require a shift away from the single-disease paradigm towards a multimorbidity framework that concurrently addresses multiple conditions and the range of risk factors (social, biological, system, environmental) associated with them [1] [2].

Modifiable risk factors for type 2 diabetes (which accounts for at least 90% of adult diabetes cases) are shared by many chronic conditions. These risk factors include: obesity, limited physical activity, hypertension and health behaviours such as smoking and unhealthy diets (https://www.heart.org/en/health-topics/diabetes/understand-your-risk-for-diabetes). The past decade has seen the development and testing of many lifestyle interventions targeting one or more of these modifiable risk factors in various settings (e.g., outpatient clinics, primary care, community-based centers). Among the most successful lifestyle interventions are the Diabetes Prevention Program (DPP) and adaptations like Look AHEAD, which have shown long-term weight loss, improvements in diabetes self-care, reductions in complications, and benefits to other health outcomes (e.g., depression, physical health-related quality of life, incontinence, sleep quality, health care use/cost) [3]-[7]. Lifestyle interventions like the DPP (and its adaptations) target constructs of social cognitive theory that support behaviour change such as observational learning, self-efficacy, reinforcement and expectations expressed by professionals or groups/peers [8]-[11]. The critical role of social cognitive theory is highlighted in a study showing that constructs from the theory explain over half the variance in participation in moderate physical activity among older adults [12]. This is consistent with an integrative review showing that interventions targeting self-efficacy and self-determination were effective in improving diabetes self-management behaviours (e.g., exercise, dietary control) and clinical outcomes [13].

However, these programs are resource intensive, motivating the development of lower-cost adaptations. These adaptations have involved a variety of settings and populations (e.g., rural adults, obese individuals, military personnel) and included: group-based delivery; virtual formats, which show promise in terms of upstream factors (e.g., attitudes, perceptions) but their ultimate impact on outcomes such as weight loss are uncertain; and fewer sessions delivered by lay professionals [14]-[17]. Studies also suggest the mode of delivery and strategies matters; for example, Befort and colleagues found that in-person group-based visits but not virtual (telephone) group-based visits resulted in statistically significant weight loss compared to in-person individual visits [14].

While the empirical evidence and theoretical foundation for intensive lifestyle interventions like the DPP are strong, less resource intensive adaptations have shown more modest or mixed results. This means the effectiveness is less certain for pragmatic applications that are more likely to be adopted and sustained in practice. There are additional gaps in the current evidence. Most studies of lifestyle interventions in the target population (older adults with diabetes) concentrate on clinical outcomes, with less consideration of other patient-oriented outcomes. For example, a recent review found that the majority of self-management interventions for adults with type 2 diabetes aimed to improve clinical outcomes such as HbA1C (83%), weight (53%), lipid profile (45%) or blood pressure (42%); whereas quality of life was considered in only 27% of the studies, and less than 16% of studies considered health literacy, satisfaction with care, or shared decision-making [18]. There is wide variation in the populations studied, limited attention to mental health, and more complex populations with diabetes are often excluded from clinical trials are and therefore less representative of those seen in practice (e.g., older adults, patients with multimorbidity) [19]. Sparse evidence exists related to interventions that address multimorbidity, consider caregiver needs, and link primary care with community-based services to address broader social determinants of health [18] [19].

Our research unit has conducted several studies of a chronic disease management and healthy lifestyle intervention known as the Aging, Community and Health Research Unit’s Community Partnership Program (hereinafter ACHRU-CPP). This 6-month self-management intervention targets older adults with diabetes and other chronic conditions and aims to address many of the existing evidence gaps, which were also identified by older adults living with diabetes. The ACHRU-CPP, underpinned by constructs from social cognitive theory [10] [11] [20] [21], was co-designed by patients, caregivers, primary care providers, and researchers. A pilot study of the ACHRU-CPP showed preliminary effectiveness (improved physical functioning) and feasibility for delivery and acceptability to clients, caregivers and providers [22]. A larger pragmatic RCT conducted in 4 sites within 2 Canadian provinces showed significant improvements in physical functioning, mental health, and diabetes self-management in the group receiving the ACHRU-CPP [23]. This work showed promising results and supported additional testing in more complex populations and diverse geographic settings and ethno-cultural populations, with the aim of further understanding the impacts of less intensive community-based diabetes and multimorbidity self-management interventions on lifestyle habits and health.

This article presents the results of the most recent clinical trial, which sought to assess the effectiveness on quality of life of the ACHRU-CPP—a low intensity 6-month community-based lifestyle intervention (additional to usual care) for community-dwelling older adults (≥ 65 years) with diabetes and at least one other chronic condition, and their caregivers. This paper presents the findings from the effectiveness evaluation; separate papers have been prepared for presentation of the findings from other portions of the research program including the implementation evaluation [24] and scalability assessment [25].

The paper was structured in accordance with the CONSORT reporting guidelines for RCTs [26].

2. Methods

2.1. Study Design

The details regarding study design and outcomes for this trial are described in the published study protocol [20], thus more briefly described in this paper. The trial was designed as a cross-jurisdictional multi-site effectiveness-implementation type II hybrid RCT, which assigned equal emphasis to the implementation and effectiveness components [27] [28]. A hybrid design was chosen because we were evaluating a complex intervention where complexity arises not only from the intervention, but also the context within which it is implemented [28]. The effectiveness component was designed to achieve the aims of comparative effectiveness research (e.g., informing practice/policy decisions, conducting the trial in real-world practice settings) [29]. To this end, we used the Pragmatic Explanatory Continuum Indicator Summary Version 2 (PRECIS-2) [30] to maximize pragmatism across the tool’s nine domains; pragmatic features of the trial included: recruitment of participants similar to those seen in practice, delivery of the trial in the practice setting by providers employed in the setting, flexibility in the delivery of the intervention similar to that seen in practice, no extraordinary follow-up measures to encourage participant engagement, and intention-to-treat principles applied in the analysis.

An important element in the current study (and all prior work on the ACHRU-CPP) was the involvement of patients and caregivers in the research itself. The study included older adult patients and caregivers who were actively involved as research partners, providing strategic guidance through their participation on the study’s Steering Committee and Community Advisory Boards at each of the trial sites to support local adaptation and implementation. They advised the research team on study design, patient-relevant outcomes, existing community assets, recruitment approaches, implementation strategies, interpretation of findings, and developing knowledge translation products to share the trial results.

2.2. Research Questions and Related Hypotheses

The research questions for the effectiveness evaluation were:

1) What is the effect of the ACHRU-CPP in addition to usual care compared to usual care alone on mental health functioning (primary outcome), physical health functioning, diabetes self-management, depressive symptoms, anxiety, social support, physical activity, basic and instrumental activities of daily living, nutrition risk (poor diet), food security, and costs of use of health services (secondary outcomes) in older adults aged ≥ 65 years with diabetes and one or more chronic condition?

2) If the intervention demonstrates a treatment effect for the primary outcome, what subgroups of older adults (e.g., sex/gender groups, those with more vs less multimorbidity) benefit most from the intervention?

3) What is the effect of the intervention compared to usual care on outcomes (health-related quality of life, depressive symptoms, anxiety, caregiver strain, costs of use of health services) of family and friend caregivers aged ≥18 years?

We hypothesized that older adult participants and caregivers in the intervention group would experience greater improvements in mental and physical functioning and other health benefits and that the intervention would be cost neutral for participants compared to usual care (consistent with results from our previous studies) [22] [23]. We note that the resulting small caregiver sample precluded making firm inferences regarding caregiver health benefits, thus the results of the caregiver analyses below should be regarded as exploratory.

2.3. Study Setting

In Canada, provincial governments have the primary responsibility for the delivery of healthcare services. Consequently, healthcare systems and services differ across the 10 provinces, and assessing Canada-wide scalability of the intervention requires testing in more than one province. The trial was conducted in two sites in each of three Canadian provinces (i.e., Ontario, Quebec and Prince Edward Island). These provinces were chosen because policy initiatives aligned with the intervention (e.g., prioritizing care for older adults and those with multimorbidity, focused on diabetes care) and the sites within them were chosen as they service a significant older adult population, reflect diversity in geography and socio-economic/cultural features, and employ providers with strong support for the intervention and availability to deliver it. Each site involved a primary care setting or diabetes (outpatient) education program, and a community partner (e.g., YMCA, community center) to collaborate with the other providers and participate in delivery of the intervention.

2.4. Study Participants

For the effectiveness evaluation we recruited two study populations: 1) older adults who received the intervention in addition to usual care versus usual care alone, and 2) caregivers of the older adults. Eligibility criteria pertaining to the older adults included: aged ≥ 65 years, diagnosed with type 1 or 2 diabetes, diagnosed with at least one other chronic condition (in addition to diabetes), enrolled in a primary care setting or diabetes education program, residing in the area (with no plans to leave during the study) served by the primary care setting or diabetes education program, capable of providing informed consent (or has a substitute decision-maker), and competent in English or French (or has an interpreter). Eligible caregivers were those who, at the time of enrollment of the older adult, were identified by that older adult as an adult (at least 18 years of age) family/friend caregiver.

Recruitment staff (existing personnel) from each of the primary care or diabetes education program sites were trained to identify potential older adult participants using medical/electronic records. They contacted potential clients by telephone to determine their interest; all eligible and interested clients were then contacted by the Research Assistants (RAs) who shared more information about the study and arranged a baseline assessment interview. During the scheduled interview, RAs obtained consent from all participants before they completed the initial (baseline) assessment. Before COVID-19, this was done in person during a home-based interview, so the participant provided signed informed consent. During COVID-19, RAs obtained verbal informed consent by phone and the consent process was audio-recorded. All data were collected by phone. Reasons for eligible participants declining to participate in the study were recorded and clients enrolled in the study were asked to invite their caregivers at the time of the baseline assessment.

2.5. Randomization Process

Participants who were eligible and agreed to participate in the study were randomly assigned to the intervention or control group using a 1:1 allocation ratio, using stratified permuted block randomization with the sequence generated by a biostatistician not involved in the recruitment process. The sites were the strata, and sequences for each site were entered into a centralized web-based randomization service (REDCap Version 11.1.9).

2.6. Intervention Arm (ACHRU-CPP and Usual Care)

The intervention was delivered at each site by an interprofessional team that included a registered nurse (RN) and a registered dietitian (RD)/nutritionist from the primary care site or diabetes education program, and a community program coordinator (PC) that was a kinesiologist or exercise specialist from the community partner organization. Managers at the primary care site or diabetes education program site recruited and selected the RN and RD/nutritionist. To avoid contamination in the control arm, all providers on the intervention team delivered the ACHRU-CPP and usual care to the intervention arm but different providers delivered usual care to participants in the control arm.

The core components of the intervention included: 1) up to three home/telephone/zoom visits by the RN and/or the RD/Nutritionist; 2) up to six monthly group sessions that include health education, exercise (gentle progressive physical activity) and informal peer support; 3) nurse-led care coordination and system navigation support provided by either the RN or RD to link participants to other health care professionals and community services as needed, ensure continuity of care across different providers and settings, and prevent gaps in care; 4) ongoing caregiver engagement and support; 5) monthly case conferences of the intervention team where the team developed and engaged in ongoing evaluation of the participant’s plan of care; and 6) collaboration with the primary care team and other specialists as needed. A care plan was developed at the initial home/telephone visit in collaboration with the participant, with documentation of the goals and actions to be undertaken, and progress was monitored in subsequent home visits. Key areas of focus for the care plan included adopting a holistic view that incorporated the social determinants of health, employing motivational interviewing strategies to foster self-efficacy, and providing self-management education and support for diabetes and their other chronic conditions. Ultimately, the core intervention components operationalized constructs in the social cognitive theory underpinning the intervention (8, 9, 19), such as building self-efficacy and self-control through peer-to-peer learning (e.g., in group sessions), receiving reinforcement from providers for behaviour change and goal attainment (e.g., in home visits where progress is regularly reviewed), and learning about and accessing additional resources (e.g., connecting participants with community resources to enable them to take control of their health and behaviours). Although concerns were shared with primary care and other specialists (with the participant’s consent), the care plan was not shared with them. Participants in the intervention arm were free to continue receiving their usual diabetes care services.

2.7. Control Arm (Usual Care Alone)

Control group participants continued to be offered usual care services through their primary care setting and/or local diabetes education program. The specific services that comprised usual diabetes care differed within and across provinces in terms of the length and focus of educational sessions, whether classes/services were required versus optional, access to on-site professionals (e.g., endocrinologist, dietitian, physiotherapist, exercise specialist, pharmacist), connections with community resources, and type/extent of follow-up services available.

2.8. Intervention Implementation Strategies

A prior protocol paper provides a detailed description of the multiple strategies used to implement and monitor delivery of the intervention [31]. Briefly these included provider training, regular outreach meetings between researchers and the intervention team to discuss progress and address challenges, and routine completion of forms documenting intervention components delivered. Compared to previous trials, implementation strategies used in this study employed virtual methods extensively to deliver intervention components. This was done in response to the pandemic restrictions but also to facilitate communication/collaboration across multiple sites and provinces.

2.9. Data Collection

Research Assistants, trained on the data collection procedures, conducted two assessments (home or telephone interview) with each study participant, one at baseline and the other immediately after the completion of the 6-month intervention period, 6 months after baseline. Baseline data were collected on socio-demographic, clinical, and primary and secondary outcome variables.

2.10. Allocation Concealment and Blinding

Participants that met the eligibility criteria and provided their informed consent were enrolled in the study and entered into the REDCap system, and were then allocated to the intervention or control group (in accordance with the randomization sequence entered into REDCap). Participants were not informed of their group allocation, although it is unlikely that they remained blinded once the intervention began. RAs conducting the baseline and 6-month interviews were blinded to group allocation, and the statistician was blinded to the group allocation. The intervention was known to the providers delivering it, however they were unaware of the study outcomes.

2.11. Outcomes

Details regarding all trial outcomes (e.g., measure, timing, analytical methods) are published in Table 1 of the study protocol [20], thus we briefly summarize them here. The primary outcome was mental functioning measured by the Mental Component Summary (MCS) score from the SF-12 (30). We chose the MCS because it is widely recognized as a quality-of-life measure that is an important patient-reported outcome (PROM). We found this measure to be responsive to the intervention in our previous RCT and consider mental functioning to be important to self-efficacy, self-management, and the behaviour change constructs embedded in the theoretical foundation of the intervention [10] [23]. We also collected data on a range of secondary outcome measures for participants, selected due to validation in older adults and anticipated responsiveness to the ACHRU-CPP:

• Physical function measured by the Physical Component Summary (PCS) score of the SF-12 [32];

• Diabetes self-management measured by the Summary of Diabetes Self-Care Activities (SDSCA) tool [33];

• Depressive symptoms measured by the Center for Epidemiological Studies on Depression 10-item scale (CESD-10) [34];

• Anxiety measured using the Generalized Anxiety Disorder 7-item scale (GAD-7) [35];

• Social support measured by the Duke Social Support Index (DSSI) [36];

• Eating and nutrition risk measured by the SCREEN II [37];

• Physical activity measured by the Physical Activity in Seniors (PASE) [38];

• Instrumental/basic activities of daily living limitations measured by the Older American Resources and Services instrument (OARS) [39];

• Shared patient-clinician decision making measured by the CollaboRATE tool [40];

• Use of health and social services measured by the Health and Social Services Use Index and research unit costing manual [41] [42].

Quality of life, self-management, mental health and activities of daily living were identified by patient/caregiver partners as important to those living with diabetes and multimorbidity. We also attempted to collect participant data on selected clinical measures (HbA1C, e-GFR, LDL-cholesterol), but COVID-19 service interruptions caused many study participants to skip or delay screening appointments. The clinical data collected from RCT participants had many missing values, uncertain timing, and/or poor alignment with study timepoints, preventing meaningful analyses/interpretation (thus no analyses of these data were performed).

For caregivers, we collected and analyzed data on caregiver strain using the Modified Caregiver Strain Index (MSI) [43], quality of life using the SF-12, depressive symptoms using the CESD-10, and anxiety using the GAD-7. These items were identified as important caregiver-reported outcome measures by patient/caregiver partners.

2.12. Statistical Analyses

The study was designed with 80% power, two-sided alpha = 0.05, and 20% attrition. It aimed to detect an effect size of 0.38 for mental functioning (primary outcome) as measured by the Mental Component Summary (MCS) score from the 12-item Medical Outcomes Study Short-Form Health Survey (SF-12), which was observed in the previous Ontario RCT [23]. The target total sample size was 264, which resulted in 88 participants per province and 44 per site [20].

The baseline demographic and clinical characteristics of participants who were randomized at baseline were summarized using descriptive statistics. Characteristics of those who dropped out compared to those that completed the study were compared descriptively. Analysis of covariance (ANCOVA) was conducted to test for group differences in the 6-month change in outcomes. Separate ANCOVA models were run for each outcome, with the 6-month outcome as the response and the baseline value of the same outcome as a covariate. The primary analyses were unadjusted for baseline differences and applied intention-to-treat (ITT) principles with participants analyzed in the groups to which they were randomized and with multiple imputation to address missing data (if substantial, e.g., >5%). Multiple imputation used multivariate imputation by chained equations with predictive-mean matching, and the imputation model included all available baseline and 6-month outcome data and covariates that were predictive of missingness. Five imputed data sets were created, the ANCOVA model was fitted to each data set, and Rubin’s rules were applied to pool model coefficients from each run [44]-[46]. Costs obtained from provincial databases were applied to the health and social service use reported by clients and caregivers [20]. These databases provide the amounts reimbursed by public and private insurance plans for insured/qualifying services, but exclude indirect and out-of-pocket patient, caregiver or productivity costs. Group differences in the 6-month change in service use costs were examined using non-parametric methods due to the skewed distributional properties of the cost data.

Pre-planned sensitivity analyses were conducted to test the robustness of the results for the primary outcome to assumptions and analytical approaches used in the primary analyses. All sensitivity analyses were conducted using complete cases. The following pre-planned sensitivity analyses were performed: a complete case analysis for comparison with the ITT results, adjusted models for significant baseline imbalances (if present), use of non-parametric methods where the parametric assumptions of the ANCOVA model were not met, and an analysis that excluded sites where participants either did not receive the intervention or did not receive it per protocol. Additionally, the COVID-19 pandemic resulted in a shift to virtual delivery of the intervention at some sites, thus a subgroup analysis exploring the treatment effect across different delivery formats (in-person, virtual, hybrid) was conducted [20].

Pre-selected factors for subgroup analyses (proposed if the treatment effect is significant for the primary outcome) were sex, number of chronic conditions, and province.

An additional unplanned sensitivity analysis was conducted, focused on “high-fidelity” only sites, in response to unplanned events that occurred during the delivery of the intervention. These events were triggered by staff shortages relating to COVID-19 responses, and the “high-fidelity” sensitivity analysis excluded sites that had to terminate the trial early due to staff redeployment, and sites that did not have a full provider team to deliver home visits or group sessions (thus may not have been able to fully address all lifestyle components of the intervention).

All statistical analyses were performed using R Version 4.4.1 (2024-06-14) and assumed a two-tailed p-value (α = 0.05). The following robust ANCOVA methods available in R were explored for outcomes where parametric assumptions were not met: WRS2 (Version 1.1 - 6, 2024-03-14), sm.ancova (Version 2.2 - 6.0, 2024-02-17) and fANCOVA (Version 0.6 - 1, 2020-11-13). The results of the robust ANCOVA models were similar for all outcomes, thus results were reported for one package (fANCOVA, which offers two statistical tests of significance: ANOVA-like statistic, variance estimator statistic).

3. Results

3.1. Participants (Baseline)

Table 1(a) provides the participant baseline characteristics. Randomization resulted in no significant group differences. Slightly more than half of participants were female (54%), and the average age was 76 years (standard deviation = 6 years). Over half of the participants (54%) were married or living with a partner and had at least a college diploma. Approximately 25% had annual household incomes above $50,000 CAD, 90% were retired, and 36% were living alone. The mean number of chronic conditions (excluding diabetes) was 5 (standard deviation = 2 conditions). Mean depressive symptom scores were approximately 5 for both groups, which is below the at-risk cut-off for the CESD-10 (32). Mean anxiety scores were approximately 2 for both groups, which is below the at-risk threshold of 8 established for the GAD-7 [35].

Table 1. (a) Baseline characteristics of RCT participants; (b) Baseline characteristics of RCT caregivers.

Table 1(b) provides the caregiver baseline characteristics and shows that the groups were relatively similar across a range of factors. For both groups, over 75% of caregivers were female, married or living with a partner, born in Canada, and white/Caucasian; the average age (groups combined) was 64 years (standard deviation = 12.7 years). Over 60% had completed at least college or some university and were retired; and 25% (groups combined) had annual household incomes above $100,000.

3.2. Attrition

Figure 1. Study flow diagram.

Figure 1 provides the study flow for participants and caregivers. Of the 839 participants assessed for eligibility, 619 (74%) met the inclusion criteria and 295 (48%) of the eligible participants accepted an invitation to join the study. Participants were randomized on a 1:1 ratio to the groups, resulting in 147 intervention and 148 control participants. Of the 295 participants randomized at baseline, 246 (83%) successfully completed the 6-month follow-up. Forty-nine (49) participants were lost to follow-up - a dropout rate of approximately 16% in each group. Reasons for refusing to participate in the study and losses to follow-up are shown in Figure 1, with the pandemic being a key reason (e.g., study stoppage, disinterest in virtual delivery). Note that complete outcome data for participants that completed the 6-month follow-up were obtained for the primary outcome (MCS) and many of the secondary outcomes (see complete case sample sizes for each group in Additional File 3), thus the dropout rate (~16%) represents the missing data rate for most outcomes.

Additional File 1 provides a comparison of those who completed the study (n = 246) compared to those who dropped out (n = 49). It shows that the two groups were similar across a range of socio-demographic and health-related characteristics; the exception was income where there was a considerably higher proportion of dropouts in the lowest annual household income category compared to completers (33% for dropouts compared to 19% for completers).

Caregivers of the 249 participants having a caregiver (84.4%) were invited to join the study, with 32 (12.9%) accepting the invitation. Caregivers were allocated to the group in which the participant was randomized, resulting in 15 caregivers in the intervention and 17 caregivers in the control group. Three (9%) caregivers were lost to follow-up, 1 in the intervention and 2 in the control group, resulting in 29 caregivers completing the 6-month data collection.

3.3. Intervention Delivery (Timing, Format, Dose)

The RCT ran from July 2019 through May of 2022, with sites starting at different times and COVID-19 impacting the sites differently depending on when the site began the trial. Additional File 2 provides the timeline for the trial by province and site. In Province 1, one site switched from in-person to virtual delivery midway through the intervention, while the other site started after the lockdown thus the entire intervention was delivered virtually. In Province 2, the intervention was running in-person but stopped at both sites shortly after the lockdown began, due to interventionists being redeployed to provide pandemic assistance. The intervention was run twice in Province 3, the first time delivered entirely in-person and the second entirely virtual.

Table 2 summarizes the intervention dose for home visits and group sessions by province and site. Home Visits: The engagement rate (% of participants receiving at least one home visit) was high, ranging from 83% to 95% across the sites. Mean home visits were approximately 2 (out of a maximum of 3). However, home visits were below the mean at Province 2 sites due to trial stoppage during the COVID-19 pandemic. Home visits were attended by either the RN or RD, except in Province 1/Site 2 where both providers attended all home visits (due to provider safety concerns). The proportion of participants receiving in-person home visits varied considerably depending on when the trial began relative to the COVID-19 lockdown, from 0% at Province 3 (both sites, as began after lockdown) to 95% at Province 2/Site 1 (prior to lockdown). Group Sessions: Mean group sessions attended by participants was 2 and 3 for Sites 1 and 2 (respectively) in Province 1, just under 2 for both sites in Province 3, and below 1 for the sites in Province 2. The proportion of participants receiving in-person group sessions varied in a pattern similar to that of home visits, from 0% for the sites in Province 3 (virtual cohort) to 73% at Province 1/Site 1.

Table 2. Intervention dose (home visits, group sessions) by province and site.

^aTrial stopped due to redeployment of interventionists to assist with COVID-19 pandemic; ^bAt most sites, one provider (Registered Nurse or Registered Dietitian) attended home visits, and the providers alternated visits. The exceptions were Province 1/Site 2 and 1 visit in Province1/Site 1, where both providers attended all home visits. Participants were offered a maximum of 3 visits, with the number of visits counted as 2 visits if attended by both providers. For this reason, the home visit count could exceed the maximum of 3 - e.g., in the case where a participant had 3 home visits and 2 providers attended all 3, this would count as 6 home visits.

Research Question 1: Intervention Effectiveness, Sensitivity Analyses (Participants)

Table 3 provides the ANCOVA model results, using multiple imputation (n = 295, n = 147 intervention & n = 148 control). No significant group differences were observed for the primary or secondary outcomes. Adjusted ANCOVA models were not run due to the absence of significant differences at baseline (see Table 1).

Table 3. Effectiveness analysis (ANCOVA model results, multiple imputation).

Various pre-planned sensitivity analyses were conducted, including:

1) Complete Case Analysis: Additional File 3 provides the results from the complete case analysis (n = 246), which is consistent with the results shown in Table 2 (no group differences for the primary or secondary outcomes).

2) Nonparametric ANCOVA: The ANCOVA model assumptions were checked; the most frequent violation was the absence of normalcy in the distribution of most outcomes, with only minor violations for a few outcomes for the assumptions of constant variance (SF-12 PCS) and equality of regression slopes (GAD, COLLABORATE). While ANCOVA is known to be robust to the main violation we observed across the outcomes (non-normality) (42), we performed nonparametric analyses for all outcomes where a violation of one or more parametric assumptions was observed. These results are included in Additional File 4 and are consistent with the parametric results.

3) “High-fidelity” sites: We separately examined the treatment effect in “high fidelity” sites, where the intervention was primarily delivered per protocol (which included virtual delivery). Two Quebec sites and the second delivery at the PEI site were excluded from this analysis because a) the two Quebec sites stopped the study (see Additional File 2) due to re-deployment of the interventionists to assist with the pandemic, and b) only one of the interventionists (dietitian) attended most home visits at the PEI site. Additional File 5 provides the results, which are consistent with the results in Table 1.

4) Intervention delivery format: We explored whether the treatment effect varied across delivery formats (all in-person, all virtual, a mix of in-person and virtual). Additional File 6 provides the results, which show that the absence of a treatment effect for all outcomes is seen across all three delivery formats.

5) Sex-differences: We performed a subgroup analysis, based on feedback from our interventionists during the trial indicating that females seemed to be more engaged in the intervention compared to males. Additional Files 7 & 8 provide the results, which show that the absence of a treatment effect for all outcomes as seen in the primary analysis was also the case for both females and males.

Table 4. Effectiveness evaluation (caregiver outcomes) (ANCOVA model results).

^aleast mean square (LSM) difference, ANCOVA model, Control - Intervention; ^bnormalcy assumption of ANCOVA model not met, fANCOVA (non-parametric) method shows mixed results: ANOVA-like statistic shows p = 0.39, variance estimator statistic shows p = 0.03; ^cnormalcy assumption of ANCOVA model not met, fANCOVA (non-parametric) method agrees with parametric results (p > 0.05 for both methods): Role Physical: ANOVA-like statistic p = 0.61, variance estimator statistic p = 0.33; Role Emotional: ANOVA-like statistic p = 0.55, variance estimator statistic p = 0.37; Mental Health: ANOVA-like statistics p = 0.41, variance estimator statistics p = 0.14; ^dall ANCOVA (parametric) model assumptions met; ^eequality of regression slopes assumption of ANCOVA not met fANCOVA (non-parametric) method agrees with parametric results (p > 0.05 for both methods): ANOVA-like statistics p = 0.32, variance estimator statistics p = 0.33; ^fnormalcy and homogeneity of variance assumptions of ANCOVA model not met, fANCOVA (non-parametric) method agree with parametric results (p > 0.05 for both methods): ANOVA-like statistic p = 0.53, variance estimator statistic p = 0.47.

Table 4 provides the results of the cost analysis, which shows that the two groups differed regarding the cost of the use of health and social services, with the result favouring the control group. The cost difference is attributed to the cost of the intervention, with the change in the costs of use of all other services being similar and negligible in both groups. The mean (sd) cost per person of the intervention was $CAD594.4 ($CAD437.6) and the median (IQR) cost was $CAD559.20 ($CAD417.4, $CAD594.30).

Research Question 2: Intervention Effectiveness, Subgroup Analyses (Participants)

No subgroup analyses were performed as no treatment effect was observed for the primary outcome.

Research Questions 3: Intervention Effectiveness (Caregivers)

As per Figure 1, 29 caregivers completed the baseline and 6-month data collection (n = 14 intervention, n = 15 control). Table 5 provides the results from the ANCOVA analysis, which showed a significant group difference for the caregiver strain index (MSI), favouring the control group (mean difference: −3.28, 95% CI: −5.63 to −0.91). As noted in the Methods above, due to the small caregiver sample, these results should be regarded as exploratory.

Cost differences in service use were not explored, as the data included only acute care service use (emergency department, hospital) and the groups were similar in reporting very few acute care events (e.g., at baseline and 6 months both groups reported one visit to the emergency department for health issues pertaining to the caregiver, and both groups reported one hospitalization at baseline and none at 6 months for the caregiver).

Table 5. Cost of use of health and social services analysis by group.

^aWilcoxon-Mann-Whitney test is a non-parametric analog to the independent samples t-test. The hypothesis being tested is whether the median differences are equal for the two groups; ^bA positive median cost difference indicates that median costs were higher at T2. A negative median cost difference indicates that median costs were higher at T1 (baseline); ^cIncludes the costs of the intervention for the intervention group, which includes the following costs: home (or virtual) visits, group wellness sessions, monthly case conferences, interventionist training. The median (IQR) cost of the intervention per patient was $559.20 ($417.4, $793.40) and the mean cost per patient was $594.40; ^eServices included in the categories: other healthcare provider services (e.g., nurse practitioner, physiotherapist, occupational therapist, speech and language pathologist, dietician, social worker, mental health counsellor, optometrist, chiropractor, dentist, pharmacist, personal support worker), alternative therapy services (e.g., naturopath), and social services (e.g., meals, homemaker, transportation).

4. Discussion

We assessed the effectiveness on quality of life of a low intensity 6-month community-based lifestyle intervention (additional to usual care) for community-dwelling older adults (≥65 years) with diabetes and at least one other chronic condition, and their caregivers. Compared to usual care, the intervention was not associated with an improvement in the primary outcome (mental functioning) or secondary outcomes (social support, health-related quality of life, functional limitations, depressive symptoms, anxiety, physical activity, nutritional risk, diabetes self-care activities, patient/provider collaboration). There are several potential hypotheses to explain the absence of results seen in the trial: 1) the pragmatic nature of the trial aimed to deliver the intervention in routine practice settings but introduced a number of challenges, 2) the pandemic-related restrictions diluted the effects of many of our core intervention components, and 3) the choice of measurement tool for the main variable may not have captured the change or the nuance in the change.

Choosing to set our trial in routine practice settings naturally made it vulnerable to numerous challenges. First, and most impactful, was the COVID-19 disruption. A separate paper presents the implementation evaluation which includes a detailed discussion of pandemic-related issues (22), thus here we only briefly highlight the concerns. Patients and caregivers on the Steering Committee and Community Advisory Boards stressed the importance of recognizing the extreme nature of the COVID-19 disruption, particularly the negative effects of the pandemic on the mental health and well-being of older adults. Their views were matched by studies showing that COVID-19 and the accompanying restrictions impacted mental health and quality of life [47] [48]. Investigators on the team also acknowledged several destabilizing effects threatening the validity of the trial results, including: the move from in-person to virtual care occurred in both groups which may have reduced the differences in care between groups (thereby diluting the treatment effect), the move from in-person to virtual occurred mid-trial and affected intervention sites differently (hence resulting in unforeseen consequences and variation in responses from both providers and participants, see Additional File 2), and the ever-changing circumstances dictated by COVID-19 led to constant change on a scale beyond that typically seen in routine practice (impacting the generalizability of the findings and stability of care and service use in both groups). Many of the threats to scientific validity that our investigators identified have been reported by others who conducted trials during COVID-19 [49]. While some argue that the RCT design offers insulation from contextual changes because both groups are affected, this ignores the unique impacts of the pandemic on this intervention which includes components that do not exist or are less intensely/consistently delivered in usual care (e.g., home visits, group sessions, system navigation/care coordination). Second, the study targeted the general population of older adults having diabetes, but not those with or at high risk of mental health problems or poor quality of life. This may have reduced the ability of the intervention to benefit those that had high baseline mental functioning or quality of life due to a ceiling effect. Table 1 shows that there were few high-risk individuals at baseline regarding mental health—e.g., average CESD and GAD scores in both groups were well below the at-risk thresholds. Additionally, Additional File 7 provides the sex-based T1 MCS mean (sd) scores for participants in our study (Females: 53.9 (9.1), Males: 53.8 (10.0)), which are similar to the norms for the Canadian population (Females: 54.6 (7.7), Males: 53.0 (8.8)) [50]. Third, our trial analyzed several secondary outcomes and performed multiple sensitivity analyses, which raises the likelihood of false-positive results. However, the consistency of our results across the large number of outcomes and analyses suggests that this is not a concern. Also, our approach aligns with recommendations for pragmatic trials, which is to include a broad range of outcomes [30]. Many of the outcomes we captured have been recommended for trials testing interventions addressing multiple chronic conditions (our participants had an average of 5 conditions and the intervention targeted management of multiple chronic conditions) [51].

As noted above, the absence of a treatment effect in our trial may be due to the pandemic-related restrictions which diluted the effects of many of our core intervention components. Significant changes resulted from the switch to virtual delivery of the intervention, including:

• home visits were impacted by reducing the number of assessments and the ability of providers to fully assess the home environment.

• group sessions were impacted by reducing the duration from 2 hours to 1 hour (to avoid online fatigue), eliminating lunches which reduced the opportunity for participants to socialize, reducing the physical activity component and limiting its tailoring, and introducing a format that did not suit all participants (due to discomfort and/or challenges with technology, preference for in-person contact) which reduced engagement and attendance.

• system navigation was challenged by the closure and/or constant state of change of many community-based services.

We searched for RCTs run during COVID-19 that were similar to ours in terms of testing patient-oriented outcomes (e.g., quality of life, self-management) in a lifestyle intervention targeting community-dwelling (non-institutionalized) older adults. Very few had been published at the time of writing, due in part to a concentration on clinical outcomes like HbA1C and hampered by inadequate descriptions of the interventions [13]. We found one RCT run during COVID-19 that tested a social needs system navigation intervention (a component in our intervention) for adults with type 2 diabetes, which showed no treatment effect for quality of life or the clinical outcomes [52]. The pandemic began in March 2020, only 6 months into our own study; at this time the literature on virtual delivery of similar interventions was limited but in general seemed to support this format, thus the research team decided to proceed with the trial using virtual delivery of the intervention [20]. Virtual delivery was not embraced by all intervention participants despite the various mitigation strategies put in place (e.g., provision of tablets, internet/wifi and technology training). Many participants in both groups struggled with virtual care, but it is important to recognize that the shift to virtual delivery in our study occurred in the early months of the pandemic when the format was new, abruptly implemented, and dramatically different from pre-pandemic, in-person care.

This shift was seen across Ontario, with one study reporting that primary care use by individuals in Ontario with type 2 diabetes declined by 17% in the first year of the pandemic with a 330% increase in those having 1+ virtual visits [53]. While some have linked reduced use of healthcare services to a decline in health status of people living with diabetes, evidence from the U.S. suggests that telehealth alleviated the negative impacts of decreased primary care screenings and risk factor management [54]. Other studies suggest that telehealth reduced the mental health impacts of the pandemic in adults living with diabetes [55]. Ultimately, virtual delivery of care in our study ensured that participants in both groups had continued access to care, which was undoubtedly a benefit to some. Moreover, virtual delivery is likely to be accepted even if not embraced in the future, since virtual care has been increasingly adopted post-pandemic and recommendations have emerged on incorporating this format into care delivery strategies across the spectrum of diabetes care [56] [57]. However, shifting to exclusively virtual delivery during the pandemic means that the optimal model of delivery for the intervention still needs to be determined. Another outstanding question is whether telehealth care will widen disparities in population subgroups managing diabetes [54]; our intervention included intensive mitigation measures (e.g., providing participants with tablets) unlikely to become part of usual care any time soon.

Another explanation of our findings may relate to challenges noted in the literature regarding moving the needle on our primary outcome. We used the SF-12 instrument (MCS score) to measure our primary outcome, which is frequently regarded as a quality-of-life measurement tool. Quality of life, while an important patient-oriented outcome, has proven to be challenging in terms of generating meaningful change for interventions that address multiple chronic conditions (a key area targeted in our intervention). A systematic review of interventions for multimorbidity (half of which focused on diabetes, depression and/or heart disease; core intervention elements included case management and multidisciplinary teams) showed little evidence of improvements in quality of life or clinical outcomes, with only modest reductions in depression for studies that targeted participants with depression [58]. Even intensive interventions such as Guided Care have not shown evidence of improving quality of life [59]. Nevertheless, intervention participants in our study reported improvements in their experience with care and positive perceptions of health benefit/impact from their care [60]. Consistent with the findings from the 3D intervention and other multimorbidity interventions such as Guided Care [59] [61], it is conceivable that the ACHRU-CPP improves perceptions and experience with care but does not improve quality of life itself [61]. Since patient experience is one of the Quadruple Aims of health care [62], perhaps this outcome should factor more prominently in an evaluation of the merits of an intervention.

Strengths and Limitations

The trial had several strengths. We used a multi-jurisdictional and multi-site approach, recruiting participants from two sites in each of three Canadian provinces. The trial was consistent with recommended guidelines for conducting pragmatic RCTs, was rigorously done with various mitigation strategies employed to address pandemic-related changes and included a range of sensitivity analyses to test the robustness of the findings to model assumptions and pandemic effects [20]. Slightly less than half (48%) of eligible participants agreed to participate in the study, which is above rates often seen in trials testing interventions in older adults with multiple chronic conditions [58] [60]. The engagement rate for core components of the intervention was high. The trial was highly pragmatic to ensure that the results were reflective of real-world implementation, e.g., we used broad eligibility criteria to ensure that the participants were representative of those seen in practice, the intervention was delivered in the real-world practice setting by providers employed in the setting, no extraordinary follow-up measures were used, patient-oriented outcomes were selected, and intention-to-treat analyses were conducted [30].

Among the key limitations experienced in this trial (other than COVID-19 mentioned above), we acknowledge the challenge in engaging caregivers in the trial (they were encouraged but not required to participate). Only 32 of 249 (12.9%) eligible caregivers accepted an invitation to the study, and 29 of the 32 completed the baseline and 6-month data collection. Consequently, the statistical analyses pertaining to caregivers should be interpreted with caution and regarded as exploratory. That is, the group difference in the change in caregiver strain favouring the control group is an unusual finding that is not explained by large differences in socio-demographic characteristics (see Table 1(b)) and may be the product of a small and unrepresentative sample (further influenced/exacerbated by pandemic-related effects). While efforts were made to blind participants (e.g., they were not advised of their group allocation), awareness of usual care services could result in awareness of membership in the intervention group and lead to bias in the self-reporting of outcomes. However, the findings and consistency of the results across a range of outcomes do not suggest a systematic response bias favouring the intervention group. Although we communicated participant concerns to their primary care or specialist team as needed, we did not share the care plan developed in the intervention. This may have reduced the ability to align/coordinate the care provided by the interventionists with the routine care participants were receiving. We acknowledge that health and service use costs may be underestimated, due to the exclusion of indirect, out-of-pocket and productivity costs. Finally, engagement rates in the group sessions at both sites in Province 2 were significantly lower compared to the other sites (see Table 2), with trial stoppage representing only a partial explanation of these low rates. Low attendance in Province 2 was also due to cultural differences (many participants preferred individual rather than group sessions) and challenges finding a suitable site that was accessible to all intervention participants.

Ethics Approval and Consent to Participate

Institutional ethics approval was obtained from the following: the Hamilton Integrated Research Ethics Board (#5101); the Scarborough Health Network Research Ethics Board (#NEP-18-014); the Unity Health Toronto Research Ethics Board (#18- 336); University of Prince Edward Island Research Ethics Board (#6008019); Prince Edward Island Research Ethics Board; and Centre intégré universitaire de santé et de services sociaux (CIUSSS) de la Capitale-Nationale (MP13-2019-1670). Ethics approval was renewed on an annual basis as required for the study duration. Informed consent was obtained from participants (older adults, caregivers, providers, managers, public and community partners) by the research assistant before study enrolment.

Availability of data and materials

The data that support the findings of this study are not openly available due to reasons of sensitivity and confidentiality, but anonymized versions of the data may be available from the corresponding author upon reasonable request.

Trial Registration

Clinical Trials.gov Identifier NCT03664583. Registration date: September 10, 2018.

Funding

This study was supported, in part, by funding from the Canadian Institutes of Health Research Strategy for Patient-Oriented Research (SPOR) Primary and Integrated Health Care Innovations Network: Programmatic Grants (Funding Reference Number: KPG-156883) in partnership with: Diabetes Action Canada, a Canadian Institutes for Health Research (CIHR) Strategy for Patient-Oriented Research Network in Chronic Disease (project reference #1.1.1ACHR); McMaster Institute for Research on Aging (Hamilton, ON); McMaster University School of Nursing; Réseau-1 Québec; Fonds de Recherche du Québec (FRQS); Scarborough Health Network Foundation. This research was also undertaken, in part, thanks to the funding from Dr. Markle-Reid’s Tier 2 CIHR Canada Research Chair and the McMaster Collaborative for Health and Aging. The funders of this study had no role in study design, data collection, data analysis, data interpretation or writing the manuscript.

Authors’ Contributions

KF prepared an initial draft of the manuscript and performed all statistical analyses. JP, MMR, RV, KF, RG and MN made significant contributions to the design and conduct of the trial as well as interpretation of the trial results. All authors read and approved the final manuscript.

Acknowledgements

We thank the older adults and caregivers who participated in this study, as well as the nurses, dietitians, nutritionists and program coordinators who delivered the intervention. We also thank the managers of intervention programs, the recruiters, research assistants and the study sites for their support of this study.

We thank the research team in the Aging, Community and Health Research Unit for supporting this study. Members of the ACHRUCPP Research Team (in addition to the co-authors of this paper): Johanne Blais, Department of Family Medicine and Emergency Medicine, Faculty of Medicine, Université Laval, Quebec City, Quebec; Andrea Gruneir, Department of Family Medicine & Dentistry, College of Health Sciences, University of Alberta, Edmonton, Alberta, Canada; Janet MacIntyre, Faculty of Nursing, University of Prince Edward Island, Charlottetown, Prince Edward Island, Canada; Angela Riveroll, Department of Applied Human Sciences, Faculty of Science, University of Prince Edward Island, Charlottetown, Prince Edward Island, Canada, Ali Ben Charif, VITAM - Centre de recherche en santé durable, Québec City, Québec, Canada; Dean Eurich, School of Public Health, University of Alberta, Edmonton, Alberta, Canada; Amiram Gafni, Department of Health Research Methods, Evidence and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; Gary Lewis, Department of Medicine and Department of Physiology, University of Toronto, Toronto, Ontario, Canada; Lynne Mansell, Patient Research Partner, Alberta, Canada; Janet Pritchard, Interdisciplinary Science and Kinesiology, Faculty of Science, McMaster University, Hamilton, Ontario, Canada; Cheryl Sadowski, Faculty of Pharmacy and Pharmaceutical Sciences, University of Alberta, Edmonton, Alberta, Canada; Diana Sherifali, School of Nursing, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; Frank Tang, Patient Research Partner, Ontario, Canada; Lehana Thabane, Department of Health Research Methods, Evidence and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; Ross Upshur, Bridgepoint Active Healthcare, Toronto, Ontario, Canada; Tyler Williamson, Centre for Health Informatics, Cumming School of Medicine and Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada.

List of Abbreviations

Additional Files

Additional File 1. Baseline characteristics of RCT completers vs dropouts.

^a DSSI = Duke Social Support Index, SCREEN II - Nutritional Risk in Older Adults Version 2, SDSCA = Summary of Diabetes Self Care Activities, SF-12 = Short Form Health Survey 12-Items, OARS = Older Americans Resources & Services, CESD = Center for Epidemiological Studies on Depression, GAD = Generalized Anxiety Disorder, PASE = Physical Activity in Seniors Scale

Additional File 2. RCT timeline by province and site.

Additional File 3. Effectiveness analysis (ANCOVA model results, complete cases).

^an, Mean (SD) for T1 outcomes are for complete cases at T2.

Additional File 4. Effectiveness analysis ANCOVA (parametric) & ROBUST ANCOVA (nonparametric) results.

^aSignificance test p-value for group variable in ANCOVA (= p-values in Additional File 1); ^bfANCOVA - a nonparametric ANCOVA procedure available in R which offers a number of different statistical significant tests (ANOVA-like and variance estimator selected); ^cAssumptions tested as follows: 1) Linearity (visual - scatterplot of covariate & outcome), 2) Equal slopes (significance of covariate x Group interaction term in ANCOVA, H0: slope = 0), 3) ~N (Shapiro Wilks test of ANCOVA model residuals, H0: residuals ~N), 4) Constant variance (Levene’s test of ANCOVA model residuals, H0: equal variances across groups ).

Additional File 5. Effectiveness analysis (ANCOVA model results, complete cases) high fidelity sites only (ONTARIO & PEI-1ST RUN).

^an, Mean (SD) for T1 outcomes are for complete cases at T2.

Additional File 6. Effectiveness analysis (ANCOVA model results, complete cases) subgroup analysis - type of intervention delivery.

^aEstimated marginal means, also known as least-squares means, and related 95% confidence intervals were generated using the emmeans package Version 1.10.5 (2024-10-14) in R; ^bGroup variable is a categorical variable that represents the three intervention delivery formats used in the RCT: 1 = Fully In-person, 2 = Hybrid, 3 = Fully Virtual; ^cANCOVA results for model that includes independent variable indicating group (intervention, control) and baseline covariate value.

Additional File 7. Effectiveness analysis (ANCOVA model results, complete cases) FEMALES (n = 133).

^an, Mean (SD) for T1 variables are for complete cases at T2.

Additional File 8. Effectiveness evaluation (ANCOVA model results, complete cases) MALES (n = 113).

^an, Mean (SD) for T1 variables are for complete cases at T2.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References