Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

Mixed method evaluation of Relational Team Development (RELATED) to improve team-based care for complex patients with mental illness in primary care

BMC Psychiatry volume 19, Article number: 299 (2019) Cite this article

Abstract

Background

Patients with mental illness are frequently treated in primary care, where Primary Care Providers (PCPs) report feeling ill-equipped to manage their care. Team-based models of care improve outcomes for patients with mental illness, but multiple barriers limit adoption. Barriers include practical issues and psychosocial factors associated with the reorganization of care. Practice facilitation can improve implementation, but does not directly address the psychosocial factors or gaps in PCP skills in managing mental illness. To address these gaps, we developed Relational Team Development (RELATED).

Methods

RELATED is an implementation strategy combining practice facilitation and psychology clinical supervision methodologies to improve implementation of team-based care. It includes PCP-level clinical coaching and a team-level practice change activity. We performed a preliminary assessment of RELATED with a convergent parallel mixed method study in 2 primary care clinics in an urban Federally Qualified Health Center in Southwest, USA, 2017-2018. Study participants included PCPs, clinic staff, and patient representatives. Clinic staff and patients were recruited for the practice change activity only. Primary outcomes were feasibility and acceptability. Feasibility was assessed as ease of recruitment and implementation. Acceptability was measured in surveys of PCPs and staff and focus groups. We conducted semi-structured focus groups with 3 participant groups in each clinic: PCPs; staff and patients; and leadership. Secondary outcomes were change in pre- post- intervention PCP self-efficacy in mental illness management and team-based care. We conducted qualitative observations to better understand clinic climate.

Results

We recruited 18 PCPs, 17 staff members, and 3 patient representatives. We ended recruitment early due to over recruitment. Both clinics developed and implemented practice change activities. The mean acceptability score was 3.7 (SD=0.3) on a 4-point Likert scale. PCPs had a statistically significant increase in their mental illness management self-efficacy [change = 0.9, p-value= <.01]. Focus group comments were largely positive, with PCPs requesting additional coaching.

Conclusions

RELATED was feasible and highly acceptable. It led to positive changes in PCP self-efficacy in Mental Illness Management. If confirmed as an effective implementation strategy, RELATED has the potential to significantly impact implementation of evidence-based interventions for patients with mental illness in primary care.

Peer Review reports

Background

Mental illness complicates care for chronic medical illness. Patients with both mental and physical illness report lower quality of life and have higher medical costs, poorer outcomes, and higher mortality rates than those with only physical illness [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]. Approximately half of treatment for psychiatric conditions in the US occurs in the primary care setting [16,17,18,19]. Therefore, improving the diagnosis and treatment of mental illness in primary care is critical to improving outcomes among patients with chronic medical illness. Team-based models of care based on the chronic care model such as the Collaborative Care Model [20,21,22,23,24] have been shown to improve outcomes for complex patients, defined here as those with both chronic medical and mental illness. However, the uptake of these models by primary care clinics has been limited and wide gaps remain in the implementation of evidence-based models of care for patients with mental illness in primary care [25].

Primary care providers (PCPs) generally express high levels of satisfaction with team-based care for patients with mental illness [26,27,28]. Transitioning to team-based care can be challenging for primary care practice [26,27,28,29,30,31,32,33,34,35,36,37,38,39,40]. Practical and psychosocial aspects of practice change influence the processes of adopting and implementing team-based care. Practical factors include the need for additional training; changes to the work space requirements; workforce availability; revised workflows; billing and reimbursement challenges; and documentation procedures.

Key psychosocial aspects of implementing practice change involve clinic climate, relational coordination, and role clarity. Clinic climate is an organizational trait that involves the presence of a shared vision of the organization; psychological safety; concern for high quality; and support of innovation [41]. Relational coordination, a theory of organizational management focusing on the interdependent relationships between people working in teams, predicts the quality of care in healthcare settings. Three relational domains — shared goals, shared knowledge, and mutual respect — overlap aspects of clinic climate but focus more directly on the interpersonal relationship between team members [42, 43]. Clinic climate and relational coordination among members of clinical teams have been shown to predict the quality of chronic disease care, and poor clinic climate can represent a significant barrier to successful practice facilitation [41, 44,45,46,47,48,49]. In implementing team-based care, clinic staff and PCPs from diverse professional backgrounds need to communicate and collaborate successfully [43, 50, 51]. However, many physicians, nurses and clinic staff are not trained in the communication skills required for high-level relational coordination. For PCPs, moving to a team-based model represents a significant change in their role in the clinical team, which can be confusing and threatening to professional self identity. PCPs can view their role within the healthcare team differently from how other team members, such as social workers and nurse practitioners, view physicians’ roles within the team, which can lead to tensions [31, 37]. Thus, role clarity (mutual understanding and respect for the expertise among members of the care team) is critical in implementation of team-based care.

Additionally, PCPs have low confidence in their knowledge and skill when caring for patients with mental illness. In previous qualitative and quantitative studies, they have expressed the need for more support in caring for complex patients with mental illness [52,53,54,55,56]. Thus, PCP’s may need focused training in the transition to a team-based approach and in the management of mental illness within a team-based approach.

A commonly used implementation strategy for team-based care is practice facilitation. Practice facilitation utilizes practice facilitators to assist clinics and healthcare systems in implementing practice change [46, 57, 58]. Practice facilitation is effective at improving communication across specialties [57, 59, 60], increasing adoption of practice change [61, 62], and building consensus [63]. Practice facilitators often have specialized training in mental illness and have been utilized in training providers in the management of mental illness [64, 65]. However, while standard practice facilitation can support implementation of team-based care models by addressing practical aspects of change, it does not explicitly address many of the psychosocial factors essential to sustainable practice change. Improved implementation strategies for team-based care should also address team climate, relational coordination, and clinician self-efficacy.

To address psychosocal factors of implementation, we developed Relational Team Development (RELATED), a brief clinic-wide intervention that combines team-level practice facilitation and PCP-level clinical coaching [66]. RELATED is an enhanced practice facilitation strategy that draws upon psychological clinical supervision models for the PCP-level clinical coaching. Psychotherapy clinical supervision has been shown to lead to increased knowledge of and increased acquisition of psychotherapy skills among both psychotherapists and psychiatric nurses [67, 68]. Further, RELATED utilizes practice facilitators with a background in clinical psychotherapy, as they have specialized training in interpersonal dynamics, communication, and conflict resolution. Table 1 illustrates the distinguishing features between RELATED and standard practice facilitiation.

Table 1 Components of Relational Team Development (RELATED) and Standard Practice Facilitation
Full size table

In this study we evaluated the feasibility and acceptability and conducted a preliminary assessment of efficacy of the RELATED intervention to address implementation barriers for team-based models of care. Although RELATED can be used to implement more general practice change, this study evaluated RELATED specifically in the implementation of team-based care for complex patients who had chronic medical and mental illness.

Methods

Study design

We utilized a convergent parallel mixed method approach to evaluate the acceptability and feasibility as well as preliminary efficacy of the RELATED intervention. The initial iterative development of the RELATED intervention is previously described [66]. We continued the iterative development of RELATED by conducting consecutive pilot studies in two clinics and making modifications to the intervention between the first and second clinic pilot.

Intervention

In RELATED, a practice facilitator with graduate training in psychotherapy utilizes a subset of psychotherapy clinical supervisory activities and standard practice facilitation techniques in a two-part intervention. RELATED includes 1) Clinical Supervision and Coaching (Coaching) in which a facilitator shadows PCPs during visits with patients who have complex needs and provides feedback on clinical care and utilization of team-based care; and 2) Practice Change Activity Team (PCAT) in which the facilitator guides the PCP participants, their clinical teams, and patient representatives as they work through an evidence-based practice change activity. Patient representatives are included in the PCAT to provide insight from the patient perspective of potential changes to clinic processes. Table 1 outlines in detail the intervention components. Team-based care and mental health didactics are delivered to PCPs, staff, leadership and patient representatives during the PCAT component. Didactics are drawn from a comprehensive educational resource including mental health and team-based care competencies that were developed by the practice facilitator and principal investigator (D.F.L). Mental health competencies include mood disorders and anxiety disorders. Team-based care competencies include components of the Chronic Care Model, Complex Leadership Theory, and quality improvement process approaches. Complex leadership theory acknowledges the complex nature of healthcare systems in which leadership emerges from the dynamic relationships between members of the system that foster the adaptive capacity of healthcare systems. Complex adaptive systems allow for the self-organized emergence of outcomes that may surpass those of a hierarchical, mechanical system [69, 70]. The practice facilitator determines which didactics are most appropriate for the group based on group dynamics and subject matter interest. Human subjects review: The study was approved by the Colorado Multiple Institutional Review Board.

Setting

We pilot tested RELATED in two primary care clinics in urban Federally Qualified Health Centers that are part of a safety net integrated healthcare system in the Southwest, USA between 2017 and 2018. Both clinics serve racially and ethnically diverse patient populations. We utilized a sequential approach to performing the intervention with time to refine RELATED processes in the period between the two clinics. The first clinic included internal medicine, family medicine, and medicine-pediatrics primary care specialties. The second clinic only included internal medicine. Both clinics had integrated psychologists and social workers on the care team providing mental illness consultation to PCPs, short-term psychotherapy to patients, and connection to the local specialty mental health system. One clinic had two full-time behavioral health providers and the other had two part-time behavioral health providers. Both clinics utilized the Primary Care Behavioral Health model of integration, offering patients access to mental health care can often be inaccessible to patients [71].

Participants and Recruitment

Participants included clinician PCPs, practice staff and patient representatives. Additionally, patients gave written consent to have their PCP visits shadowed during the Coaching component. Currently practicing PCPs (MDs/DOs, Nurse Practitioners, and Physician Assistants) were recruited for the full intervention. PCPs were recruited in a face-to-face meeting during their regularly scheduled provider meeting supplemented by follow up emails. Resident physicians were excluded from the study due to schedule limitations. For the Coaching component, patient visits were identified by the practice facilitator and approved for shadowing by the PCP. Patients were approached prior to their visit by the practice facilitator for informed consent to have their visit shadowed. Patients were eligible if they met our definition of complexity—patients with at least one mood or anxiety disorder and one chronic medical illness. Patients were excluded if they were unable to consent due to language other than Spanish and English or cognitive delay. Clinic staff and leadership were recruited for the PCAT. Eligible clinic staff members included nurses, medical assistants, social workers, behavioral health providers, patient navigators, addiction counselors, clinic leadership, and front desk staff. The intervention was explained by the study team in a regularly scheduled all-clinic meeting, and the clinic leadership decided which staff would be invited to participate in the intervention. In addition, the practice facilitator, who treats patients in a different primary care clinic within the healthcare system, helped identify and contact eligible patient representatives using a phone script to see if they were interested in participating in the PCAT portion of the intervention. Patients were contacted by the practice facilitator up to three times. All participants signed written consent.

Evaluation

We used a convergent parallel mixed method approach to the evaluation [72, 73]. Our primary outcomes were feasibility and acceptability. Secondarily we assessed preliminary efficacy of the RELATED intervention. We used quantitative and qualitative methods as complementary evaluation means to gain a more complete assessment of RELATED (Additional file 1 for reporting standards). We used quantitative surveys to assess intervention acceptability and to evaluate changes in PCP measures pre- and post- intervention (Additional file 2 for PCP survey). Qualitative focus groups helped provide a more nuanced perspective on the acceptability and feasibility of the intervention. Finally, field notes collected by the research team helped assess clinic climate to complement the survey and focus group results.

Data collection

Primary outcomes

Primary outcomes were acceptability and feasibility of RELATED. Acceptability was measure through an acceptability survey and focus groups. Feasibility was measured in ease of recruitment and implementation of RELATED.

Secondary outcomes

We surveyed participating PCPs pre- and post-intervention to assess change in self-efficacy in mental illness management and team-based care; communication self-efficacy; knowledge of mental illness management; attitudes toward team-based care; and team climate.

Further, field notes from observations were used to document daily clinic routines and workflows, document interactions among staff and between staff and PCPs, and to gain a better understanding of the overall clinic climate.

Survey instruments

We adapted the Healthcare Provider Acceptability of a Behavioral Intervention to Promote Adherence to apply to RELATED [74]. It consisted of 4 items and is scored on a 4 point Likert scale from “completely disagree” to “completely agree”. Questions addressed included: time involved in intervention, setting for intervention, appropriateness of intervention, and impact of intervention.

The PCP pre- and post-intervention survey consisted of several validated measures. We previously developed and validated measures of PCP self-efficacy in mental illness management and team-based care [75, 76]. The mental illness management (10 items) and team-based (7 items) scales are scored on a 0 to 10 Likert scale where 0 is “not at all confident” and 10 is “extremely confident”. We modified validated instruments to measure changes in communication self-efficacy, knowledge of mental illness management, attitude toward team-based care and team climate [77,78,79,80,81]. We chose a limited number of items from each instrument using a combination of published factor loadings and content domains. Communication was assessed with a modified version of the Communication Skills Self-assessment, which is focused on provider-patient communication [77]. The four-point response scale was modified to the 11-point scale used for the team-based care and mental illness management self-efficacy scales. The self-assessment was reduced from 35 questions to 11 questions. To assess provider attitudes, we selected five items from the 26 item Attitudes toward Health Teams Scale [78]. Likewise, to assess team climate we used six items of 61 from the Team Climate Inventory which focused on clinical team function and communication. Higher scores indicated more positive attitudes toward team-based care and team climates. We constructed summary variables for the following domains: Communication Skills Self-Assessment, Attitudes Toward Health Teams Scale and Team Climate Inventory by calculating the mean score for each set of items within the scale. Mental health treatment knowledge was assessed with a version of Katerndahl’s instrument that assesses mental health treatment for major depressive disorder (MDD), generalized anxiety disorder (GAD), and panic disorder adapted to include bipolar disorder. Participants were asked to rate how effective (not effective, somewhat effective, effective, or don’t know) each of 14 different psychotropic medications were for MMD, GAD, and bipolar disorder. Higher scores indicated higher mental illness treatment knowledge [81]. These modifications were made and initially utilized in a state-wide survey of Colorado PCPs. These methods were previously described [75, 76].

Survey procedures

Surveys to assess acceptability were administered on paper to participating PCPs and clinical staff at 1 and 6 months after the start of the intervention. The PCP survey was administered in person prior to the start of the intervention and after the intervention. PCP surveys took approximately 10 minutes to complete. All surveys were de-identified.

Focus groups

We conducted three focus groups at each site. Staff and patients were included in the same focus group because the patients were integrated into the PCAT along with staff. Separate focus groups were conducted with PCPs and clinic leadership to encourage the members of each group to feel safe in expressing themselves. The focus groups were semi-structured to allow for a comprehensive and systematic, yet flexible approach [82]. We assessed acceptability and feasibility of the intervention; the effects of the intervention on team dynamics; sustainability of the intervention; and suggested changes to the intervention. The focus groups were 60 minutes each; they were conducted by research assistants; lunch was served as an incentive. The focus groups were audio recorded and transcribed. In focus groups in the first clinic, we elicited feedback on suggested changes to improve RELATED.

Clinic climate

In addition to focus groups, we utilized a modified ethnographic approach to better understand clinic climate [83]. We used less intrusive methods for primary data collection and limited methods that would require provider or staff time away from patients. Data collection included field notes by the practice facilitator and site coordinator. The same research assistants who help conduct the focus groups also shadowed staff in workflow observation. In each clinic at least one clerk, medical assistant, nurse, behavioral health specialist, and social worker were shadowed for a half-day.

Analysis

Quantitative data were analyzed using SAS version 9.4 software. All statistical tests were performed with a level of significance of 0.05. Self-efficacy subscales, as well as attitudes toward team-based care, team climate, and acceptability scales were scored by calculating mean of responses across all items in the patient care domain for each provider. Knowledge of effective treatments of mental illness subscales were scored as the percent of correct answers in each domain for each time period. Pre-post evaluations of the patient care domains were performed with paired t tests.

For both the focus groups and clinic observations, we used an iterative and team-based process guided by qualitative content analysis, and data were collected until saturation of themes was reached [84, 85]. Two qualitatively trained analysts both inductively and deductively developed a code book. Initial domains for codes were based on the interview guide domains and the codebook was expanded based on codes that emerged from the data. The analysts jointly reviewed and coded the dataset of focus group transcripts and observation field notes until no new codes were identified and there was strong code assignment agreement. The practice facilitator provided member checking throughout code book development. All transcripts were independently read, double coded, and then merged prior to analysis. Any discrepancies in coding were addressed through discussion and consensus among the coders. Throughout the analytic process, the analytic team (qualitative analysts) met regularly with the broader study team to check new findings, discuss emergent new codes and themes, and assess the preliminary and final results. ATLAS.ti version 8.0 was used for data organization and management.

Results

We recruited 18 PCPs for the full intervention in the two clinics. We only sent one recruitment email to PCPs at each clinic, as we met or exceeded our goal of eight per clinic following this initial invitation. PCPs were predominantly female, middle age and non-Hispanic White. We recruited 12 physician and six non-physician PCPs. Fifty-eight patients agreed to have their visits shadowed for the coaching component. For the PCAT only, we recruited 20 clinic staff (including non-PCP leadership) and 3 patient representatives in the two clinics. The PCPs recruited for the full intervention also participated. Staff were primarily female and White and about a third Hispanic ethnicity. About half of the patients whose visits were shadowed in the Coaching component identified as Hispanic. Many patients indicated race as “other” or did not indicate (Table 2). All PCPs and staff completed the study. One patient representative in the PCAT was withdrawn due to concern raised by their PCP about PCAT participation. PCAT attendance averaged 63% and 67% among PCPs and staff, respectively, in clinic one and 75% and 65% for PCPs and staff, respectively, in clinic two. One patient completed 5 of 6 of the PCAT sessions and completed the focus group. One patient completed 3 of 6 of the PCAT sessions and did not participate in the focus group.

Table 2 Primary Care Provider, Patient, and Staff Demographics
Full size table

Staff and PCP participants found the intervention highly acceptable. Mean scores on the acceptability measure at one month were 3.8 (Standard Deviation = 0.3) on a 1-4 Likert scale with N=36. Mean scores at six months after the start of the intervention were 3.7 (SD = 0.4) with N= 33. All 18 PCPs completed the pre- and post- intervention surveys. Although not designed to power for changes in the pre-post surveys, there was a statistically significant improvement in PCP mental illness management self-efficacy and in knowledge in treating major depressive disorder (Table 3). Scores are reported as combined between two clinics. There was no significant difference between the two clinics.

Table 3 Pre-post Changes in PCP Survey Scores
Full size table

Focus group results complemented and reinforced the findings from the survey concerning acceptability and feasibility. In focus groups, the majority of participants described RELATED as valuable, efficient, and rewarding. PCPs overwhelmingly found the coaching sessions helpful and suggested adding more sessions. One PCP described feeling burdened by the time spent in the PCAT sessions. Leadership and staff had positive comments about the PCAT and, notably, did not express concerns about the time commitment. Staff and PCPs described positive effects of RELATED on team dynamics, inclusivity and patient-centeredness. It was also noted that RELATED supported a patient-centered approach by bringing the patient perspective to the forefront of practice change. Finally, RELATED was found to help level the traditional medical hierarchy, opening opportunities for staff previously not engaged in leadership behaviors to adopt them. See Table 4 for a summary of focus group results.

Table 4 Focus Groups Results
Full size table

In Clinic One focus groups, we also elicited suggested changes from PCPs and staff to inform the second pilot intervention in Clinic Two. PCPs and staff suggested several changes to the intervention. PCPs recommended changes to maximize their coaching time with the practice facilitator, such as scheduling complex patients when the practice facilitator was present; more shadowing sessions with the practice facilitator; and blocking off the last hour of their patient schedule for feedback from the practice facilitator. PCPs suggested changes to the feedback session, including more constructive feedback; educational materials offered during the sessions; and provision of information on best practices in communication. Regarding the PCAT component, PCPs suggested blocking off patient care time for the meetings and using graphics to track the practice change activity. They requested additional topics for the didactics, including opioid dependency, bipolar disorder, substance use disorders, personality disorders, and managing medications. Both PCPs and staff suggested inviting more staff to participate in the PCAT; creating ongoing follow-up on the practice change to check in on progress, separating the didactics from the PCAT and making the clinical didactics optional for staff. Clinic staff also suggested involving clinic leadership from the beginning.

Research staff shadowed 17 staff members from clinic one and eight from clinic two. Team interactions in PCAT sessions were also observed. Observations at the two pilot clinics highlighted key features of clinic climate that could impact practice change. Clinic one typically acted as a single unit; PCPs and staff displayed a “we” approach to caring for patients and there was cohesion among staff and providers/leadership. Clinic staff and PCPs displayed high levels of trust with each other. Clinic two appeared to function within the traditional medical hierarchy and emphasized individual performance. Both clinics demonstrated mostly positive work relationships and a significant amount of support among staff.

We made several adaptations to RELATED prior to clinic two based on feedback from clinic one: 1) The scheduling of the Coaching component was changed to mirror the consultation model used by the behavioral health providers in the clinics; the practice facilitator was available to Coach the PCPs from one clinical pod per clinic session rather than a single PCP per session. 2) Coaching feedback sessions were shortened to 5-7 minutes after each patient observation; 3) Brief didactic interventions were incorporated into the Coaching component; 4) A booklet was created for participants to track PCAT progress; 5) The time between the 5th and 6th PCAT sessions was increased to address any issues that arose in initial implementation of the practice change; 6) Clinic leadership was invited to join the PCAT from the beginning; 7) Additional staff were invited to join the PCAT; and 8) The mental health didactics were changed to mandatory for PCPs but optional for staff and patient representatives. After these changes were made and piloted in clinic two, no further suggestions were made by clinic two participants to change the intervention.

Discussion

RELATED is a novel approach to implementing team-based approaches to care for mental illness in primary care settings. In this pilot, we demonstrate that RELATED is both feasible and highly acceptable among PCPs, clinic staff, and clinic leadership. Both quantitative and qualitative assessments of acceptability were positive. In fact, PCPs requested additional Coaching sessions. Focus group data was consistent across staff, PCPs and leadership, citing overall improvement in team functioning, including inclusivity across roles and a better understanding of the interdependence of others’ roles. We experienced minimal barriers to study recruitment or logistics of the intervention. Our primary feasibility challenge was incomplete attendance for the PCAT sessions. Although not designed to be powered for the PCP survey outcomes, PCPs had a significant increase in their mental illness management self-efficacy and improvement in their knowledge of treatment of major depression in pre- and post-intervention surveys. Although features of the clinic climate differed in the two pilot clinics, we found no significant differences in acceptability or feasibility results. We also found no difference in PCP survey results across clinics. We did not find a change in Attitudes Toward Team-based Care or Clinic Climate measures.

RELATED represents a promising new method for implementing practice change in the primary care setting, specifically for the care of patients with mental illness and chronic medical disease. Although RELATED builds on standard practice facilitation, it has key innovations. RELATED has a unique emphasis on improving PCP engagement in the care of patients with mental illness and with the team-based approach to care employed in their clinic. It increases engagement by improving PCP self-efficacy in the management of complex patients in the context of the healthcare team. This increased engagement is theorized to lead to practice change with more depth and sustainability than traditional practice facilitation. PCPs have shared discomfort with treating mental illness due to a lack of knowedge and training. They also have expressed poor self-efficacy in communicating with patients with mental illness [52, 55, 56]. By addressing these deficits, PCPs are given the opportunity to more effectively engage in the interdisciplinary care of their most complex patients.

In addition to the practical aspects of practice change, RELATED addresses both PCP-level and clinic-wide interpersonal relationships. RELATED utilizes a practice facilitator with specialized training in health psychology as well as traditional practice facilitation techniques. This background facilitates the clinical supervision of PCPs during the Coaching component of RELATED and the psycho-therapeutic training provides practice facilitators with enhanced skills to manage relational coordination dynamics during the PCAT. Although clinical supervision has been shown to increase knowledge of and acquisition of psychotherapy skills in diverse professional roles [67, 68], such supervision is not included in standard practice facilitation. Standard practice facilitation often involves coaching a practice team through an evidence-based quality improvement project. The PCAT is innovative in that it serves multiple purposes in RELATED. It is a vehicle for teaching the team the basic skills of practice change. Additionally, it serves as a lab for the team to improve their Relational Coordination skills and for PCPs to improve their self-efficacy in team-based care. These changes are intended to decrease barriers to care and practice change activities moving forward.

Despite the different features of clinic climate observed at each clinic, we were both surprised and encouraged that acceptability and PCP survey results did not differ between the two clinics. RELATED works directly with clinic climate in its focus on PCP communication skills and clinic interpersonal relationships. Although additional research is needed, this early finding may point to RELATED’s success in working with diverse clinic climates in the implementation of practice change. However, since we did not find changes in the Attitudes Toward Team-based Care or Clinic Climate measures, the effect of RELATED on these domains needs further evaluation. Both the positive and null findings need further confirmation.

The importance of clinic climate on practice change has been illustrated in studies of the patient centered medical home (PCMH). In a literature review of factors affecting implementation of intensive outpatient care in Veterans’ Affairs medical homes, Breland, et.al. found that factors in the inner clinical setting were key to success of the interventions including a culture of innovation, good systems for communication, clinic leadership engagement, and positive tension for change (i.e., the belief that there was a need for the change). They also noted characteristics of staff and leadership in the successful clinics, which included creativity, flexibility and strong interpersonal skills. Clinicians noted the importance of “a cohesive team of very skilled individuals” [86]. These findings are supported by a study of key facilitators and barriers to the implementation of PCMH in nine Minnesota practices, where clinic climate was seen as a key factor. The alignment of values with the practice change and openness to innovation were facilitators, while clinics with resistance to change and providers who felt they did not need the change were noted as barriers to implementation [87]. By targeting clinic climate, RELATED may significantly improve the implementation of practice change in primary care. RELATED’s focus on PCP-specific role clarity, clinic-wide interpersonal relationships, and clinic climate could all be applied to the implementation of a variety of practice change initiatives for chronic disease in general.

As a pilot, this study has multiple limitations. Our primary outcomes were acceptability and feasibility. Though we did evaluate effectiveness in PCP self-efficacy, communication, knowledge, attitudes and team-climate, we were not powered for those measures. We also did not evaluate RELATED for its effectiveness in implementing a specific practice change. As a next step, we plan to evaluate its effectiveness in implementing measurement-based care for anxiety and depression in primary care. The study was performed in clinics from one urban federally qualified, safety net integrated delivery system in the Southwestern United States. Therefore, the results are not necessarily generalizable to other types of primary care practices or other regions. We plan to conduct further trials in diverse clinic settings to test the generalizability of these findings. The intervention would need to be adapted to utilize external mental health resources for clinics without integrated behavioral health. Further, with testing in only two clinics, our findings on the effect of clinic climate, feasibility, acceptability, and PCP survey results can only be seen as generative. It will be important to further explore the ability of RELATED to impact implementation of team-based care with larger studies. In order to scale-up RELATED, we envision healthcare systems and clinics using the tools acquired in an intervention for the implementation of future practice change activities. Further research on the sustainability of the effects of RELATED would be necessary to test this hypothesis. RELATED is more resource intensive than traditional practice facilitation. Cost effectiveness will need to be evaluated prior to taking it to scale. This will include factors suggesting the additional cost of RELATED over traditional practice facilitation is warranted (e.g., clinics with a history of failed practice change efforts). Additionally, the use of practice facilitators with graduate training in psychotherapy is somewhat restrictive. However, given the increasingly widespread implementation of integrated behavior health in primary care, a workforce intimately familiar with the nuances of mental health care delivery in primary care is already in place that could be trained to deliver RELATED. We also plan a formal evaluation of the effect of RELATED on clinic Relational Coordination. The evaluation was performed by the research team, who were known to participants. Thus, these results may be subject to social desirability bias. Lastly, we had no control group to evaluate change in the PCP measures. In the pre- post design, PCPs are only controlled against their initial scores. This design is subject to selection bias, as only PCPs who volunteered to participate were evaluated.

Conclusion

We successfully piloted the RELATED implementation strategy. RELATED was feasible and highly acceptable. It led to positive changes in PCP self-efficacy in mental illness management and team based care. Next steps involve conducting a larger trial to test the effectiveness of RELATED in implementing a specific evidence-based practice change to improve the clinical care of patients with mental illness in primary care. If confirmed to be effective as a strategy, the potential applications include training practice facilitators to work healthcare systems where they could assist individual practices with multiple practice change activities within a continuous practice improvement model. Individual practice facilitators could also be trained to work with multiple different small practices, and returning for follow-up practice change activities as needed. As a novel implementation strategy, RELATED has the potential to advance the field of practice change for patients with mental illness in primary care.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

PCPs:

Primary Care Providers

RELATED:

Relational Team Development

PCAT:

Practice Change Activity Team

PCMH:

Patient Centered Medical Home

References

  1. Rochon PA, Katz JN, Morrow LA, et al. Comorbid illness is associated with survival and length of hospital stay in patients with chronic disability. A prospective comparison of three comorbidity indices. Med Care. 1996;34(11):1093–101.

    Article  CAS  PubMed  Google Scholar 

  2. Librero J, Peiro S, Ordinana R. Chronic comorbidity and outcomes of hospital care: length of stay, mortality, and readmission at 30 and 365 days. J Clin Epidemiol. 1999;52(3):171–9.

    Article  CAS  PubMed  Google Scholar 

  3. Fortin M, Bravo G, Hudon C, et al. Relationship between multimorbidity and health-related quality of life of patients in primary care. Qual Life Res. 2006;15(1):83–91.

    Article  PubMed  Google Scholar 

  4. Michelson H, Bolund C, Brandberg Y. Multiple chronic health problems are negatively associated with health related quality of life (HRQoL) irrespective of age. Qual Life Res. 2000;9(10):1093–104.

    Article  CAS  PubMed  Google Scholar 

  5. Wolff JL, Starfield B, Anderson G. Prevalence, expenditures, and complications of multiple chronic conditions in the elderly. Arch Intern Med. 2002;162(20):2269–76.

    Article  PubMed  Google Scholar 

  6. Glynn LG, Valderas JM, Healy P, et al. The prevalence of multimorbidity in primary care and its effect on health care utilization and cost. Fam Pract. 2011;28(5):516–23.

    Article  PubMed  Google Scholar 

  7. Starfield B, Lemke KW, Bernhardt T, Foldes SS, Forrest CB, Weiner JP. Comorbidity: implications for the importance of primary care in 'case' management. Ann Fam Med. 2003;1(1):8–14.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Egede LE, Nietert PJ, Zheng D. Depression and all-cause and coronary heart disease mortality among adults with and without diabetes. Diabetes Care. 2005;28(6):1339–45.

    Article  PubMed  Google Scholar 

  9. Katon W, Fan MY, Unutzer J, Taylor J, Pincus H, Schoenbaum M. Depression and diabetes: a potentially lethal combination. J Gen Intern Med. 2008;23(10):1571–5.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Connerney I, Sloan RP, Shapiro PA, Bagiella E, Seckman C. Depression Is Associated With Increased Mortality 10 Years After Coronary Artery Bypass Surgery. Psychosom Med. 2010.

  11. Hamer M, Batty GD, Stamatakis E, Kivimaki M. The combined influence of hypertension and common mental disorder on all-cause and cardiovascular disease mortality. J Hypertens. 2010.

  12. Barth J, Schumacher M, Herrmann-Lingen C. Depression as a risk factor for mortality in patients with coronary heart disease: a meta-analysis. Psychosom Med. 2004;66(6):802–13.

    Article  PubMed  Google Scholar 

  13. Kreyenbuhl J, Dickerson FB, Medoff DR, et al. Extent and management of cardiovascular risk factors in patients with type 2 diabetes and serious mental illness. J Nerv Ment Dis. 2006;194(6):404–10.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Slomka JM, Piette JD, Post EP, et al. Mood disorder symptoms and elevated cardiovascular disease risk in patients with bipolar disorder. J Affect Disord. 2012;138(3):405–8.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Perron BE, Howard MO, Nienhuis JK, Bauer MS, Woodward AT, Kilbourne AM. Prevalence and burden of general medical conditions among adults with bipolar I disorder: results from the National Epidemiologic Survey on Alcohol and Related Conditions. J Clin Psychiatry. 2009;70(10):1407–15.

    Article  PubMed  Google Scholar 

  16. Wang PS, Lane M, Olfson M, Pincus HA, Wells KB, Kessler RC. Twelve-month use of mental health services in the United States: results from the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):629–40.

    Article  PubMed  Google Scholar 

  17. Mental disorders in general medical practice an opportunity to add value to healthcare. Behav Healthc Tomorrow. 1996;5(5):55-62, 72.

  18. Katon W, Schulberg H. Epidemiology of depression in primary care. Gen Hosp Psychiatry. 1992;14(4):237–47.

    Article  CAS  PubMed  Google Scholar 

  19. Kessler RC, Demler O, Frank RG, et al. Prevalence and treatment of mental disorders, 1990 to 2003. N Engl J Med. 2005;352(24):2515–23.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Wagner EH, Austin BT, Von Korff M. Improving outcomes in chronic illness. Managed care quarterly. 1996;4(2):12–25.

    CAS  PubMed  Google Scholar 

  21. Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. The Milbank quarterly. 1996;74(4):511–44.

    Article  CAS  PubMed  Google Scholar 

  22. Bodenheimer T. Interventions to improve chronic illness care: evaluating their effectiveness. Dis Manag. 2003;6(2):63–71.

    Article  PubMed  Google Scholar 

  23. Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness: the chronic care model, Part 2. JAMA. 2002;288(15):1909–14.

    Article  PubMed  Google Scholar 

  24. Katon WJ, Lin EH, Von Korff M, et al. Collaborative care for patients with depression and chronic illnesses. N Engl J Med. 2010;363(27):2611–20.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Mechanic D. More people than ever before are receiving behavioral health care in the United States, but gaps and challenges remain. Health Aff (Millwood). 2014;33(8):1416–24.

    Article  Google Scholar 

  26. Farrar S, Kates N, Crustolo AM, Nikolaou L. Integrated model for mental health care. Are health care providers satisfied with it? Canadian family physician Medecin de famille canadien. 2001;47:2483–8.

    CAS  PubMed  PubMed Central  Google Scholar 

  27. Younes N, Passerieux C, Hardy-Bayle MC, Falissard B, Gasquet I. Long term GP opinions and involvement after a consultation-liaison intervention for mental health problems. BMC Fam Pract. 2008;9:41.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Kilbourne AM, Greenwald DE, Bauer MS, Charns MP, Yano EM. Mental Health Provider Perspectives Regarding Integrated Medical Care for Patients with Serious Mental Illness. Administration and policy in mental health. 2011.

    Article  Google Scholar 

  29. Kisely S, Duerden D, Shaddick S, Jayabarathan A. Collaboration between primary care and psychiatric services: does it help family physicians? Canadian family physician Medecin de famille canadien. 2006;52:876–7.

    PubMed  Google Scholar 

  30. Gallo JJ, Zubritsky C, Maxwell J, et al. Primary care clinicians evaluate integrated and referral models of behavioral health care for older adults: results from a multisite effectiveness trial (PRISM-e). Ann Fam Med. 2004;2(4):305–9.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Franx G, Oud M, de Lange J, Wensing M, Grol R. Implementing a stepped-care approach in primary care: results of a qualitative study. Implementation science : IS. 2012;7:8.

    Article  PubMed Central  Google Scholar 

  32. Katon W, Von Korff M, Lin E, Simon G. Rethinking practitioner roles in chronic illness: the specialist, primary care physician, and the practice nurse. Gen Hosp Psychiatry. 2001;23(3):138–44.

    Article  CAS  PubMed  Google Scholar 

  33. St Peter RF, Reed MC, Kemper P, Blumenthal D. The scope of care expected of primary care physicians: is it greater than it should be? Issue brief. 1999;24:1–4.

    Google Scholar 

  34. St Peter RF, Reed MC, Kemper P, Blumenthal D. Changes in the scope of care provided by primary care physicians. N Engl J Med. 1999;341(26):1980–5.

    Article  CAS  PubMed  Google Scholar 

  35. Wagner EH. The role of patient care teams in chronic disease management. BMJ. 2000;320(7234):569–72.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Wright B, Lockyer J, Fidler H, Hofmeister M. Roles and responsibilities of family physicians on geriatric health care teams: Health care team members' perspectives. Canadian family physician Medecin de famille canadien. 2007;53(11):1954–5.

    PubMed  PubMed Central  Google Scholar 

  37. Leipzig RM, Hyer K, Ek K, et al. Attitudes toward working on interdisciplinary healthcare teams: a comparison by discipline. J Am Geriatr Soc. 2002;50(6):1141–8.

    Article  PubMed  Google Scholar 

  38. Sommers LS, Marton KI, Barbaccia JC, Randolph J. Physician, nurse, and social worker collaboration in primary care for chronically ill seniors. Arch Intern Med. 2000;160(12):1825–33.

    Article  CAS  PubMed  Google Scholar 

  39. Steiner JL, Ponce AN, Styron T, Aklin EE, Wexler BE. Teaching an interdisciplinary approach to the treatment of chronic mental illness: challenges and rewards. Academic psychiatry : the journal of the American Association of Directors of Psychiatric Residency Training and the Association for Academic Psychiatry. 2008;32(3):255–8.

    Article  Google Scholar 

  40. Coleman K, Reid RJ, Johnson E, et al. Implications of reassigning patients for the medical home: a case study. Ann Fam Med. 2010;8(6):493–8.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Bower P, Campbell S, Bojke C, Sibbald B. Team structure, team climate and the quality of care in primary care: an observational study. Qual Saf Health Care. 2003;12(4):273–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Gittell JH, Beswick J, Goldmann D, Wallack SS. Teamwork methods for accountable care: relational coordination and TeamSTEPPS(R). Health Care Manag Rev. 2015;40(2):116–25.

    Article  Google Scholar 

  43. Gittell JH, Fairfield KM, Bierbaum B, et al. Impact of relational coordination on quality of care, postoperative pain and functioning, and length of stay: a nine-hospital study of surgical patients. Med Care. 2000;38(8):807–19.

    Article  CAS  PubMed  Google Scholar 

  44. McHugh M, Brown T, Liss DT, Walunas TL, Persell SD. Practice Facilitators’and Leaders’ Perspectives on a Facilitated Quality Improvement Program. Ann Fam Med. 2018;16(Suppl 1):S65–71.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Noel PH, Romero RL, Robertson M, Parchman ML. Key activities used by community based primary care practices to improve the quality of diabetes care in response to practice facilitation. Qual Prim Care. 2014;22(4):211–9.

    PubMed  PubMed Central  Google Scholar 

  46. Van der Wees PJ, Friedberg MW, Guzman EA, Ayanian JZ, Rodriguez HP. Comparing the implementation of team approaches for improving diabetes care in community health centers. BMC Health Serv Res. 2014;14:608.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Cramm JM, Nieboer AP. In the Netherlands, rich interaction among professionals conducting disease management led to better chronic care. Health Aff (Millwood). 2012;31(11):2493–500.

    Article  Google Scholar 

  48. Deri Armstrong C, Taljaard M, Hogg W, Mark AE, Liddy C. Practice facilitation for improving cardiovascular care: secondary evaluation of a stepped wedge cluster randomized controlled trial using population-based administrative data. Trials. 2016;17(1):434.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Liddy C, Hogg W, Singh J, et al. A real-world stepped wedge cluster randomized trial of practice facilitation to improve cardiovascular care. Implement Sci. 2015;10:150.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Gittell JH, Godfrey M, Thistlethwaite J. Interprofessional collaborative practice and relational coordination: improving healthcare through relationships. J Interprof Care. 2013;27(3):210–3.

    Article  PubMed  Google Scholar 

  51. Cramm JM, Nieboer AP. Relational coordination promotes quality of chronic care delivery in Dutch disease-management programs. Health Care Manag Rev. 2012;37(4):301–9.

    Article  Google Scholar 

  52. Loeb DF, Bayliss EA, Binswanger IA, Candrian C, de Gruy FV. Primary care physician perceptions on caring for complex patients with medical and mental illness. J Gen Intern Med. 2012;27(8):945–52.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Loeb DF, Bayliss EA, Candrian C, de Gruy FV, Binswanger IA. Primary care providers’ experiences caring for complex patients in primary care: a qualitative study. BMC Fam Pract. 2016;17(1):1–9.

    Article  Google Scholar 

  54. Loeb DF, Binswanger IA, Candrian C, Bayliss EA. Primary care physician insights into a typology of the complex patient in primary care. Ann Fam Med. 2015;13.

    Article  PubMed  Google Scholar 

  55. Oud MJ, Schuling J, Slooff CJ, Meyboom-de JB. How do General Practitioners experience providing care for their psychotic patients? BMC Fam Pract. 2007;8:37.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Ballester DA, Filippon AP, Braga C, Andreoli SB. The general practitioner and mental health problems: challenges and strategies for medical education. Sao Paulo medical journal = Revista paulista de medicina. 2005;123(2):72–6.

    Article  PubMed  Google Scholar 

  57. Nagykaldi Z, Mold JW, Aspy CB. Practice facilitators: a review of the literature. Fam Med. 2005;37(8):581–8.

    PubMed  Google Scholar 

  58. Nutting PA, Crabtree BF, Stewart EE, et al. Effect of facilitation on practice outcomes in the National Demonstration Project model of the patient-centered medical home. Ann Fam Med. 2010;8 Suppl 1:S33-S44; S92.

    Article  PubMed  Google Scholar 

  59. Mold JW, Peterson KA. Primary care practice-based research networks: working at the interface between research and quality improvement. Ann Fam Med. 2005;3(Suppl 1):S12–20.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Astrop P. Facilitator--the birth of a new profession. Health visitor. 1988;61(10):311–2.

    CAS  PubMed  Google Scholar 

  61. Hogg W, Baskerville N, Nykiforuk C, Mallen D. Improved preventive care in family practices with outreach facilitation: understanding success and failure. Journal of health services research & policy. 2002;7(4):195–201.

    Article  Google Scholar 

  62. Geboers H, van der Horst M, Mokkink H, et al. Setting up improvement projects in small scale primary care practices: feasibility of a model for continuous quality improvement. Qual Health Care. 1999;8(1):36–42.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  63. Baskerville NB, Hogg W, Lemelin J. Process evaluation of a tailored multifaceted approach to changing family physician practice patterns improving preventive care. The Journal of family practice. 2001;50(3):W242–9.

    CAS  PubMed  Google Scholar 

  64. Knox L, Taylor E, Geonnotti K, et al. Developing and Running a Primary Care Practice Facilitation Program: A How-to Guide (Prepared by Mathematica Policy Research under Contract No. HHSA290200900019I TO 5.) AHRQ Publication No. 12-0011. Rockville, MD: Agency for Healthcare Research and Quality.; December 2011.

  65. Bashir K, Blizard B, Bosanquet A, Bosanquet N, Mann A, Jenkins R. The evaluation of a mental health facilitator in general practice: effects on recognition, management, and outcome of mental illness. The British journal of general practice : the journal of the Royal College of General Practitioners. 2000;50(457):626–9.

    CAS  Google Scholar 

  66. Loeb DF, Kline DM, Kroenke K, et al. Designing the relational team development intervention to improve management of mental health in primary care using iterative stakeholder engagement. BMC Fam Pract. 2019;20(1):124.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Holloway EL, Neufeldt SA. Supervision: its contributions to treatment efficacy. J Consult Clin Psychol. 1995;63(2):207–13.

    Article  CAS  PubMed  Google Scholar 

  68. Bradshaw T, Butterworth A, Mairs H. Does structured clinical supervision during psychosocial intervention education enhance outcome for mental health nurses and the service users they work with? J Psychiatr Ment Health Nurs. 2007;14(1):4–12.

    Article  CAS  PubMed  Google Scholar 

  69. The Health Foundation. Evidence Scan: Complex Adaptive Systems. The Health Foundation. http://www.health.org.uk/publication/complex-adaptive-systems. Published 2010. .

  70. Lipsitz LA. Understanding health care as a complex system: the foundation for unintended consequences. JAMA. 2012;308(3):243–4.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  71. Reiter JT, Dobmeyer AC, Hunter CL. The Primary Care Behavioral Health (PCBH) Model: An Overview and Operational Definition. J Clin Psychol Med Settings. 2018;25(2):109–26.

    Article  PubMed  Google Scholar 

  72. Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Serv Res. 2013;48(6 Pt 2):2134–56.

    Article  PubMed  PubMed Central  Google Scholar 

  73. Zhang W, Creswell J. The use of “mixing” procedure of mixed methods in health services research. Med Care. 2013;51(8):e51–7.

    Article  PubMed  Google Scholar 

  74. McLean KA. Healthcare Provider Acceptability of a Behavioral Intervention to Promote Adherence. Open Access Theses, University of Miami; 2013.

  75. Loeb DF, Crane LA, Leister E, et al. Development and initial validation of primary care provider mental illness management and team-based care self-efficacy scales. Gen Hosp Psychiatry. 2017;45:44–50.

    Article  PubMed  Google Scholar 

  76. Loeb DF, Leister E, Ludman E, et al. Factors associated with physician self-efficacy in mental illness management and team-based care. Gen Hosp Psychiatry. 2018;50:111–8.

    Article  PubMed  Google Scholar 

  77. Ashbury FD, Iverson DC, Kralj B. Physician Communication Skills: Results of a Survey of General/Family Practitioners in Newfoundland. Med Educ Online. 2001;6:1.

    Article  Google Scholar 

  78. Heinemann GD, Schmitt MH, Farrell MP, Brallier SA. Development of an Attitudes Toward Health Care Teams Scale. Evaluation & the health professions. 1999;22(1):123–42.

    Article  CAS  Google Scholar 

  79. Anderson NR, West MA. Measuring climate for work group innovation: development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235–58.

    Article  Google Scholar 

  80. Kivimaki M, Elovainio M. A short version of the Team Climate Inventory: Development and psychometric properties. J Occup Organ Psychol. 1999;72(2):241–6.

    Article  Google Scholar 

  81. Katerndahl D, Ferrer RL. Knowledge About Recommended Treatment and Management of Major Depressive Disorder, Panic Disorder, and Generalized Anxiety Disorder Among Family Physicians. Primary care companion to the Journal of clinical psychiatry. 2004;6(4):147–51.

    Article  PubMed  PubMed Central  Google Scholar 

  82. Patton MQ. Qualitative Evaluation Methods. Beverly Hills: Sage Publications, Inc.; 1980.

    Google Scholar 

  83. Bunce AE, Gold R, Davis JV, et al. Ethnographic process evaluation in primary care: explaining the complexity of implementation. BMC Health Serv Res. 2014;14:607.

    Article  PubMed  PubMed Central  Google Scholar 

  84. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

    Article  PubMed  Google Scholar 

  85. Stemler S. An Overview of Content Analysis. Pract Assess Res Eval. 2001;7:17.

    Google Scholar 

  86. Breland JY, Asch SM, Slightam C, Wong A, Zulman DM. Key ingredients for implementing intensive outpatient programs within patient-centered medical homes: A literature review and qualitative analysis. Healthc (Amst). 2016;4(1):22–9.

    Article  Google Scholar 

  87. Fontaine P, Whitebird R, Solberg LI, Tillema J, Smithson A, Crabtree BF. Minnesota’s Early Experience with Medical Home Implementation: Viewpoints from the Front Lines. J Gen Intern Med. 2015;30(7):899–906.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Angela Moss for assisting in the data analysis of the quantitative data from the implementation of the intervention. We would also like to thank Dr. Michael Fisher for his assistance in the qualitative analysis. In addition, we would like to thank Rossana Blanco Prado for her contributions to the implementation of the pilot, conducting and assisting of focus groups, and collecting clinic culture data.

Contributions to literature

The lessons learned herein will apply broadly.

  • RELATED is an enhanced practice facilitation implementation strategy that has the potential to improve the care of complex patients through a brief intervention that combines practice facilitation and psychology clinical supervision methodologies.

  • RELATED has the potential to significantly impact outcomes for mental illness in primary care by improving implementation of evidence-based models of care.

  • The RELATED strategy could be modified for use in implementing team-based care strategies for other common chronic conditions in primary care.

Funding

National Institute of Mental Health (NIMH) 5K23MH100162 and University of Colorado Health System Research Funds funded this study.

Author information

Authors and Affiliations

  1. Division of General Internal Medicine, University of Colorado School of Medicine, Academic Office 1; Mailstop B180; 12631 East 17th Ave., Aurora, CO, 80045, USA

    Danielle F. Loeb, Cori Depue, Danielle M. Kline & Dixie G. Good

  2. Lowry Family Health Center, Denver Health, Denver, CO, USA

    Samantha Pelican Monson

  3. Adult and Child Consortium for Health Outcomes Research and Delivery Science (ACCORDS), University of Colorado, Aurora, CO, USA

    Steven Lockhart

  4. Kaiser Permanente Washington Health Research Institute, Seattle, WA, USA

    Evette Ludman

  5. Department of Family Medicine, University of Colorado, Aurora, USA

    Donald E. Nease Jr

  6. Kaiser Permanente Colorado Institute for Health Research, Aurora, CO, USA

    Ingrid A. Binswanger & Elizabeth A. Bayliss

  7. Department of Family Medicine, University of Colorado School of Medicine, Aurora, CO, USA

    Frank V. de Gruy

Authors
  1. Danielle F. Loeb
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Samantha Pelican Monson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Steven Lockhart
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cori Depue
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Evette Ludman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Donald E. Nease Jr
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ingrid A. Binswanger
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Danielle M. Kline
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Frank V. de Gruy
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Dixie G. Good
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Elizabeth A. Bayliss
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

DL served as the PI of the study project; made substantial contributions to conception and design of research study; was involved with the interpretation of the data; and drafted the manuscript. SM made substantial contributions to conception and design of research study; was involved with the interpretation of the data; served as the practice facilitator for the intervention, and helped revise the manuscript. SL led the qualitative analysis of the focus groups and clinic culture, and made substantial contributions to the manuscript. CD helped with the qualitative analysis of the focus groups and clinic culture data. EL, DN, IB, FD, and EB all substantially and equally contributed to the conception and design of research study; were involved with the interpretation of the data; and helped to revise the manuscript. DK conducted focus groups, collected qualitative data when observing the clinic culture, assisted in the data analysis of the research study and interpretation of the data; and was involved with revising the manuscript. DG conducted focus groups, collected qualitative data when observing the clinic culture, assisted in the qualitative data analysis and was involved with revising the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Danielle F. Loeb.

Ethics declarations

Ethics approval and consent to participate

Informed consent was obtained from all participants in the study. Patients who had their visits shadowed by the practice facilitator provided informed written consent. PCPs, clinic staff, clinic administrators, patient respresentitives who participated in Coaching, PCAT, and/or focus groups provided written informed consent. Clinic staff who were observed in the clinic climate observation research were given an information sheet and provided verbal consent. Verbal consent, rather than written consent, was obtained to avoid workflow disruptions. Verbal consent was approved by the institutional review board as the risks were deemed minimal. The Colorado Institutional Review Board at the University of Colorado (Aurora, Colorado) approved all study procedures.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1:

Standards for Reporting Implementation Studies: the StaRI checklist for completion. Reporting standards for implementation.

Additional file 2:

PCP survey. Pre- and Post- PCP survey.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Loeb, D.F., Monson, S.P., Lockhart, S. et al. Mixed method evaluation of Relational Team Development (RELATED) to improve team-based care for complex patients with mental illness in primary care. BMC Psychiatry 19, 299 (2019). https://0-doi-org.brum.beds.ac.uk/10.1186/s12888-019-2294-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12888-019-2294-1

Keywords

BMC Psychiatry

ISSN: 1471-244X

Contact us