The Context of Services:
Identifying a Service Delivery Model and Testing its Variations

Personnel	Study Methods
Problem and Background	Key Findings
Conceptual Model	Outputs
Research Questions	References
	Latest Updates

Personnel

Nancy M. Koroloff, Ph.D., Principal Investigator
Janet Walker, Ph.D., Project Collaborator
Kathryn Schutte, M.A., Research Assistant

Problem and Background

Systems of care for children's mental health are complex entities, made even more so by the fact that service delivery dimensions exist across multiple levels (Hernandez, et al. 2001). While many studies have been done to establish the evidence base for specific interventions at the child and family level there is little research that looks at the impact of the context of services upon the delivery of these model interventions. One of the more popular service delivery approaches is team-based Individualized Service/Support Planning (ISP) also known as Wraparound. According to Faw (1999), as many as 200,000 children and their families have been served through ISP/Wraparound approaches-particularly those children who are most in need and thus most vulnerable. The philosophy of ISP/Wraparound, the ISP planning process and the challenges to its implementation are described on the web page of another RTC research project, ISP/Wraparound Teamwork in Practice. This project focuses on the organizational and system context of ISP, examining what conditions need to be in place for ISP to thrive and be efficiently practiced at the team level.

There is fairly consistent agreement in services research that service systems are comprised of at least three levels that interact to create the entirety of the system, (Rosenblatt, 1998; Miles, Franz & Donner, 1998; Amado & McBride, 2002). It is our premise that each of these levels and their interaction must be considered in an analysis of what contributes to effective ISP teamwork. We use the terms 1) system level, 2) organizational level, and 3) team level. Definitions of these three levels follow:

The system level (or policy and funding context) denotes the larger service policy and economic context that surrounds the teams and team members' agencies. The system level is made up of multiple organizations that may focus on a specific set of services (e.g. mental health), a geographic area (e.g. county), population (e.g. children) or a combination of these. The system level may also include multiple governmental entities at the county, region or state as well as other organizations that set policy, monitor or enforce policy or interpret state or national policies to local service providers. The system level also includes any body that has been constructed to oversee the development of the service system or to manage funds that have been pooled. The system context varies from community to community but at the very least will include those individuals and bodies that make decisions regarding policies and procedures and the allocation of resources that affect the functioning of the lead organization(s) and by extension, the teams.

The organizational level refers to those agencies and programs involved in supporting ISP teams in their work. We find it useful to distinguish between two roles that organizations or agencies can play relative to ISP teams. In the first role, an agency serves as "host" to the teams and is responsible for hiring, training, and supervising team facilitators. In some cases, this entity may also provide training for team members. In the second role, an agency acts as a "partner" to the team-based ISP process by contributing services, flexible funds and/or staff who serve as team members. Communities have developed a variety of strategies for distributing these roles across different agencies.

The team level refers to the set of teams that may exist in a particular community, with each team specifically constituted to serve an individual child and family. There is general agreement that an ISP team should include the caregiver and youth and at least two or three other consistently attending core members who may be representatives of key service agencies or may be family members, friends or other community supporters. The team is led by a trained facilitator and is charged with creating and implementing plans to meet the needs of the family and child with an emotional disorder. This core team may be supplemented as necessary by others who attend when their role in the plan is under consideration or when their input is invited.

• to develop a conceptual model of the necessary conditions at organizational and system level.
• to use this conceptual model to develop an assessment tool that will allow team members to assess organizational support.
• to use this conceptual model to develop an assessment tool that will allow program managers and supervisors to assess system level support.
• to verify these assessment tools against the reality of ISP practice in community sites.
• to explore approaches to feeding the results of the assessments back to decision makers.
• to catalogue the most commonly identified system and organizational barriers and solutions.

Conceptual Model

The work in this project and that of the ISP/Wraparound Teamwork in Practice project is based in a conceptual model developed during the first phase of work for the two projects. This conceptual framework was developed through a process of "backward mapping" (Elmore, 79/80; Friedman, 1999). Backward mapping begins with a description of the desired behavior at the lowest level of intervention—in this case the team level—and then proceeds to identify the necessary resources and supports that must be provided at other levels if the desired behaviors are to occur. The conceptual framework is found in Figure 1 (this figure can be found at the bottom of this section) and serves to identify the conditions that are necessary at organizational and system level for effective ISP to occur. Five themes have been delineated from our review of the literature and our interviews with expert team members and national experts in ISP/Wraparound. These five themes are 1) the practice model, 2) collaboration/partnerships, 3) capacity building/staffing, 4) acquiring services/supports and 5) accountability. For each of these five themes we have identified the necessary conditions at each of the three levels. This complex three by five matrix has been supported with our own research and the research of others and verified by expert review.

The conceptual framework described here does not prescribe the arrangement of components and proposes that the necessary conditions for the implementation of high quality ISP teams may be met even in the absence of a developing system of care. In fact we have seen ISP teams function successfully within a wide range of variation in the extent to which they receive organizational and system support. It appears, however, that different configurations of support have implications for the viability of individual teams, the stresses experienced by various stakeholders in the teams, and the sustainability of ISP programs over time. What is more, while some isolated teams may function well in the absence of organizational and system support that meets the proposed necessary conditions, we do not believe that high-quality programs will be able to do so.

For example, at the level of least support from either organizations or systems, we have observed some teams which function for extended periods of time independently of any ISP program. These independent teams are unsupported by any formal arrangements at the organizational or system level. We have also seen evidence of the potential for ISP programs to be successful within systems that were almost indifferent to their existence. Usually, such programs are operated using what we call a single agency model of an ISP program. (This similar to the agency model described in Franz, 2002). In the single agency model, the ISP program exists within an established, well-regarded human service agency which is able to provide strong support as the lead agency for ISP. Outside of this strong lead agency, the necessary conditions for high quality ISP (i.e. the conditions fulfilled by partner organizations and the larger system) are met in a minimal way, and often through informal agreements or special arrangements.

Most teams and programs appear to exist in a context of somewhat higher levels of support as in the developing system of care. Often there are formal interagency agreements recognizing teams and providing pools of funds that can be used flexibly, as well as interagency committees which meet to problem solve or to create policies supportive of ISP teamwork. In general, however, ISP programs with tenuous, newly-developing and/or only nominal system support appear to be quite vulnerable to turnover among system-level allies and to changes in funding arrangements. Such programs are often funded under pilot agreements or grant-based initiatives, and their support may wane quickly once the trial period ends.

Figure 1. Necessary Conditions

Research Questions

The specific research questions addressed by this project are as follows:

• Does the conceptual model describing necessary conditions at the organizational and system level match the understanding and experience of ISP team members, program managers, supervisors and system level staff?
• Do the assessments of organizational and system conditions have face validity when judged by program managers, supervisors, system staff and team members?
• To what extent do the necessary conditions at organizational and system level exist in selected community sites?
• Does the identification of a lack of organizational and system support in specific areas lead to effective problem solving and resolution of these barriers?
• Do different levels of organizational and system support seem to be associated with the effectiveness of teams, the stresses experienced by teams, and/or the sustainability of ISP programs over time?
• Can sites be characterized as providing ISP through independent teams, a single agency, or a developing system of care?
• Do different configurations of support have implications for the effectiveness of teams, the stresses experienced by teams, and the sustainability of ISP programs over time?

This research systematically examines the extent to which certain organizational and system supports are in place and contributing to effective ISP teamwork. Families are generally highly supportive of ISP/wraparound as a model of service delivery. However, lack of organizational and system level supports may lead to teams that are poorly trained and supervised or too stressed to practice effectively, inadequate implementation of ISP plans, lack of resources to pay for or develop needed services and lack of accountability to assure that families and children receive effective ISP.

Study Methods

Phase 1: Model development and validation (all activities completed).

Literature review and model development. Conceptual work for this project was done in conjunction with the ISP/Wraparound teamwork in practice project and began with a literature search for relevant sources on organizational and system level research. The focus was on locating research literature that identified aspects of organizations or systems that contribute to effective service provision. Literature from a number of fields was reviewed, including health, mental health, child welfare, as well as literature from education and business. Personal contacts were made with other researchers studying organizational and system concepts.

Expert interviews. We conducted semi-structured interviews with 44 individuals identified as associated with well functioning ISP programs. Interviewees were nominated as most expert within service systems receiving national recognition for best practice in ISP. Sixteen individuals were interviewed who had experiences as an ISP supervisor, trainer, program manager or associated with the system level supporting outstanding ISP. Twenty-eight interviews were conducted with expert team members, each of whom had worked with multiple teams in roles that included facilitator, care coordinator, resource developer, and parent partner/advocate. The semi-structured interviews lasted about an hour each, and focused on interviewee perceptions of factors that influenced the success or failure of ISP teams. The information from these interviews was combined with the results of the literature review to develop the first draft of the conceptual framework.

Expert review. A document describing the conceptual framework was reviewed in a group discussion format by seven experts in the field of children's mental health representing stakeholders from the service, organization, and system levels. Following revisions, 11 additional reviewers with high levels of expertise regarding ISP critiqued the document during individual feedback sessions. This group of reviewers represented the service, organization, and system perspectives and included family members, practitioners and administrators from diverse backgrounds. Revisions were made, and three assessments were developed and added to the document (see Phase II). Two of the assessments examine ISP implementation at the organization and system levels. The document and assessments then underwent another review from the first group. Final revisions were then made to the document and the assessments (Walker et al., 2003). After these revisions, the conceptual framework was formally reviewed and adopted by a group of 28 nationally recognized experts in ISP/wraparound as part of a framework to guide the "Wraparound Model Definition Initiative" described in the ISP/Wraparound Teamwork in Practice project.

Of the total 45 expert reviews of both framework and assessments (see Phase II), twelve were given by parents, four by youth or young adult consumers, ten by researchers, eight by ISP facilitators or care coordinators, five by state level administrators, five by system-level administrators, six by ISP program administrators, and two by consultants. (This total is greater than 45 due to reviewers in multiple roles relative to ISP teams and programs.) Among the 45 reviews, seven were provided by African Americans, three by Native Americans, and three by Latinos. The rest of our reviewers were Caucasian, or their ethnicity was unknown.

Phase II. Organizational and system level assessment development and review (all activities completed)

Development of assessments. Grounded in the conceptual framework and the data collected during reviews of the framework, we developed two assessments. The assessments were designed with an eye toward issues of mutual accountability across the various levels of implementation of ISP and are seen as tools for upward accountability. Neither of these assessments are intended to provide a rating or ranking for organizations or systems. Neither are they sufficiently developed to be used to measure change over time.

The Assessment of Organizational Supports (AOS) assesses the necessary conditions at the organizational level from the perspective of team members looking upward. Each of the sections of the AOS focuses on one of the five conditions listed at the organizational level in Figure 1. For each condition, the AOS lists a series of features that index the extent to which the condition is in place. Individuals completing the AOS provide two ratings for each feature, the extent to which the feature is in place and the level of priority assigned to improvement of the feature. The AOS is designed to be completed by team members who participate on several teams and who have a sense of whether or not a feature is consistently in place across more than one team.

The Assessment of the Assessment of the Policy and Funding Context (APFC) also employs an "upward accountability" strategy. Respondents for this system-level assessment include program managers, supervisors, and/or agency administrators. As with the AOS, each section of the assessment focuses on one of the conditions listed in Figure 1. Respondents use the same dual rating system described for the AOS.

Review of the assessments. The assessments were added to the document describing the conceptual framework about half way through the review process. Therefore, most of the expert review of the assessments is described under Phase I. In addition to interviews with experts, we also conducted two "reaction groups" a process similar to focus groups but with a less structured analysis. These two groups were conducted during a grantee meeting for the Comprehensive Children's Mental Health System of Care Initiative. One group involved invited program and system administrators and ISP supervisors (11 individuals) and the second group included ISP facilitators, parent partners and trainers (12 individuals). These individuals came from a variety of geographic locations throughout the United States. During the session, we asked the individuals to review and comment on each item, the totality of items in each section, and the response format. Based on these comments, the assessments were revised.

Phase III: Pilot testing of two assessments (activities 40% completed)

The two assessments will be pilot tested in at least 4 sites during Phase III. The first two sites have been identified and data has been collected at both. The process of feeding the results back to the sites is underway. These two sites are local and data were collected face-to-face during staff meetings. This allowed us to debrief the participants after they completed the assessments and obtain their reaction to the items. The assessments are being prepared for completion via the Internet. The next two pilots will be conducted with sites in other parts of the country and individuals will log onto a password protected website to complete the assessments. We have identified at least 8 sites that would be willing to serve as pilots for the web-based application.

Key Findings

• The conceptual model describing necessary conditions at the organizational and system level has been validated by two sources. The conceptual model is consistent with the experience of ISP team members, program managers, supervisors and system level staff interviewed and those in the focus groups. It has also been reviewed and accepted by a group of nationally recognized experts on ISP.
• The two assessment instruments have passed the review for face validity by ISP practitioners. The Assessment of Organizational Support and the Assessment of the Policy and Funding Context have been reviewed at the item level by individuals associated with good ISP practice and judged to fairly represent the features of organizational and system context.
  

Outputs

Publications completed: Together with staff from the ISP/Wraparound Teamwork in Practice project, we have published a monograph describing the necessary conditions to support high quality ISP/Wraparound at the team, organization and system levels. You can download this monograph by going to the publications page, http://www.rtc.pdx.edu/pgPublications.php, and searching by using the author field and entering "Walker." The citation for this monograph is:

• Walker, J.S., Koroloff, N. & Schutte, K. (2003). Implementing high-quality collaborative Individualized Service/Support Planning: Necessary conditions. Portland, OR: Research and Training Center on Family Support and Children's Mental Health, Portland State University.

We have also produced two bibliographies. One is a research bibliography listing key ISP/Wraparound references, entitled "Individualized Service/Support Planning (ISP) and Wraparound: Research Bibliography." The second bibliography is a practice-oriented bibliography listing various trainings and tools you can use to improve your work. This bibliography is entitled "Individualized Service/Support Planning (ISP) and Wraparound: Practice-Oriented Resources." You can download these bibliographies by going to the publications page, http://www.rtc.pdx.edu/pgPublications.php and searching by using the author field and entering "Walker."

We have produced several articles for the national research review, Focal Point. One article is on teamwork, another article addresses the necessary conditions needed to support effective ISP/Wraparound, and two short articles are based on interviews with a youth on a Wraparound/ISP team and family member who is also a Wraparound/ISP facilitator. These articles are cited below. You can access this Focal Point issue by going to the Focal Point page at: http://www.rtc.pdx.edu/pgFocalPoint.shtml and click on the Fall 2003 issue.

• Koroloff, N., Schutte, K., & Walker, J. S. (2003). The context of services: The importance of agency and system support to effective ISP/Wraparound. Focal Point: A National Bulletin on Family Support and Children's Mental Health, 17(1), 8-11.
• Schutte, K. (2003). Cody's experience of wraparound. Focal Point: A National Bulletin on Family Support and Children's Mental Health, 17(1), 16.
• Walker, J. S. (2003). Practices to promote effective teamwork in ISP/Wraparound. Focal Point: A National Bulletin on Family Support and Children's Mental Health, 17(1), 12-14.
• VanWormer, R., & Schutte, K. (2003). Wearing two hats: The challenges and strengths of being a parent and a wraparound facilitator. Focal Point: A National Bulletin on Family Support and Children's Mental Health, 17(1), 29.

Lastly, we have completed two assessment tools, the Assessment of Organizational Support (AOS) and the Assessment of the Policy and Funding Context (APFC). These assessments were described above and can be found in the monograph.

Publications anticipated: We plan to present at 3 national conferences and one statewide conference in collaboration with the staff from Project 6. We will submit proposals to 2 other national conferences. We also plan to write an article to be submitted to a peer-reviewed journal focused on the conceptual framework at the organizational and system level. A second peer-reviewed article will focus on the assessments and the results of the pilot test. We intend to produce 2 non-technical brochures about the two assessment tools to be used during training and presentations. Lastly, we will also develop a system of individualized profiles to give feed back on the results from the assessment tools to each pilot site.

Presentations, training, and technical assistance completed: We have presented results from our research at four national conferences. Two of these presentations were invited plenary presentations. The majority of these presentations have been done with a parent of a child with a serious emotional/behavioral challenge and with staff from the ISP/Wraparound teamwork in practice project. We have presented the assessments to three agency staff meetings and one system level staff meeting. These presentations are linked in the Latest Updates section.

Presentations, training and technical assistance anticipated: Presentations have been accepted at three national conference (in conjunction with the ISP/Wraparound Teamwork in Practice project) and we anticipate presenting at least two additional national conferences and a national training institute. The assessments will be presented to at least two more agency staff meetings and at least on more system level staff meeting. Results of the pilot tests will be reported to at least 4 pilot sites and the interpretation of these results and recommendations for next steps will be provided along with other technical assistance.

References

• Amado, A.N. and McBride, M.W. (2002) Realizing individual, organizational, and systems change: Lessons learned in 15 years of training about person-centered planning and principles. In Person-centered planning: Research, practice, and future directions (Holburn, S. and P.M. Vietze, eds.),pp. 361-377, Paul H. Brookes.
  
• Elmore, R.F. (1979/80) Backward mapping: Implementation research and policy decisions. Political Science Quarterly 94,601-650.
  
• Faw, L. (1999). The state wraparound survey. In Systems of care: Promising practices in children's mental health, 1998 series: Volume IV. Promising practices in wraparound for children with severe emotional disorders and their families (Burns, B.J. and Goldman, S.K., eds.) pp. 79-83, Center for Effective Collaboration and Practice, American Institutes for Research.
  
• Franz, J. (2002) Building the caring enterprise. http://www.paperboat.com/calliope/enter1.html
  
• Friedman, R.M. (1999). A conceptual framework for developing and implementing effective policy in children's mental health. Research and Training Center for Children's Mental Health, Department of Child and Family Studies, The Louis de la Parte Florida Mental Health Institute, University of South Florida.
  
• Glisson, C., & Hemmelgarn, A. (1998). The effects of organizational climate and interorganizational coordination on the quality and outcomes of children's service systems. Child Abuse &Neglect, Vol. 22, No. 5, 401-421.
  
• Johnsen, M. C., Morrissey, J.P., & Calloway, M.O. (1996). Structure and Change in Child Mental Health Service Delivery Networks. Journal of Community Psychology, 24, No.3, 275 289.
  
• Miles, P., Franz, J., & Donner, R. (1999) Statewide wraparound pilot project implementation principles and guidelines. Unpublished.
  
• Walker, J. S., Koroloff, N., & Schutte, K. (in press). Implementing high-quality collaborative individualized service/support planning: Necessary conditions. Portland OR: Research and Training Center on Family Support and Children's Mental Health.

Latest Updates

If a site wants to ensure quality implementation of the process of team-based individualized services planning (ISP, also often known as wraparound), what conditions must be present at the team, organization, and system levels? Project staff have derived a conceptual framework describing these conditions, and have developed a series of tools to measure whether these conditions are present at the practice, organizational, and system levels. To view the report and the tools, use the link below to "Implementing high quality Individualized Service/Support Planning..."

This project completed its research in the fall of 2004, having pilot tested assessments in three sites around the country. New reports generated from the project's work will continue to appear over throughout 2005 and perhaps beyond. Use the link below to search the RTC's publications database for the latest products. Search using "Walker" and/or "Koroloff" as authors.

The National Wraparound Initiative also continues with work on this topic. Use the link below to access the NWI home page.