APCDs currently operate in thirteen states, with five more being implemented [2] . In broad terms, an APCD’s purpose is to facilitate curtailing the rising costs of healthcare [1] [3] [4] [5] [6] [7] . APCDs provide transparency in pricing across healthcare providers [7] . This transparency, which is reasoned, will allow market forces, instigated by the payers and the public, to drive down cost and increase quality [3] [4] [5] . Transparency, however, involves more than listing prices for services. Analytic capacity, which provides actionable information, is also necessary [5] . For example, providing the cost to consumers for a maternity episode should include all costs for all providers and all care settings from the first prenatal visit to the last postnatal visit. However, developing actionable information requires stakeholder participation to determine the use cases that should be addressed in a given location to provide value to consumers for useful decision-making and the costs to be included [3] [4] .

We present an evaluation process that stakeholders viewed favorably. Stakeholders, such as employers, the public, insurers, and payers, all participate in healthcare decisions. All are influential drivers in cost reduction [5] . Public health researchers at universities and the Utah Department of Health (UDOH), and policy makers are also interested in quality of care and in optimal use of APCDs [3] . Gathering and engaging a stakeholder pool with diverse interests is important to develop a useful APCD [1] [6] . Other successful evaluation efforts have noted that ongoing and systematic evaluation, including development and use of logic models, is also important [8] [9] [10] . Identifying the process of change is a key component of our evaluation program. Because of the critical nature of evaluation questions as part of the evaluation plan, we developed the questions iteratively with stakeholders.

Utah has had an APCD since September 2009, with 21 health insurance carrier plans submitting enrollment, pharmacy, and medical file data as early as 2007 [11] . The Health Data Committee (HDC) statute authorizes data collection and provides strategic and policy oversight of Utah’s healthcare data systems, including the APCD [12] . The Utah Partnership for Value Driven Healthcare (UPV), a statewide community collaborative hosted by Health Insight (the Quality Improvement Organization for Utah), provides recommendations to the HDC about analyses and enhancements to improve functioning of the Utah APCD [13] . Stakeholders external to UDOH’s Office of Healthcare Statistics (OHCS) participate in the governance structures. These stakeholders include consumers, providers, policy makers, researchers, and payers, as well as organizations related to health information exchange and healthcare quality improvement [12] .

The All Payer Claims Database Development Manual (2015) suggests that several components are essential for APCD development [1] . Based on early learning about APCD development, these areas include engagement, governance, funding, technical build, analyses and application development [2] . APCDs evolve over time and funding may come from a variety of sources. The most recent funding resulted from the CMS Grants to States to Support Health Insurance Rate Review and Increase Transparency in Health Care Pricing, Cycle III (Cycle III grant) available through the UDOH. The Cycle III grant required an evaluation component for which we developed a novel APCD evaluation process that builds on recommendations found in the APCD Development Manual [1] .

Before the Cycle III grant award, the OHCS had a limited ability to undertake needed analyses to disseminate meaningful information, including price transparency, to facilitate healthcare reform efforts. While OHCS hosted the APCD in SQL databases―where authorized users can access data files and perform analyses using statistical software such as SAS or Stata―OHCS was dependent upon vendors to produce standard reports. For example, the APCD did not incorporate information on insurance premiums, rates, benefits, risk adjustment, or quality metrics. The lack of these data reduced the utility of the available information to inform consumer choices about healthcare services.

Figure 1 illustrates the proposed APCD improvements. The improvements described in the figure included plans to fulfill the three aims of the Cycle III grant. In Aim 1, data collection would be enhanced. In Aim 2, the analytic capacity of the Utah APCD would be expanded, and in Aim 3, dissemination of APCD results would be improved.

We used a Model for Collaborative Evaluation (MCE) built upon six precepts utilized in unity with stakeholders throughout the project: 1) Identify the situation; 2) Clarify expectations; 3) Establish a shared commitment; 4) Ensure open communication; 5) Encourage best practices; and 6) Follow specific guidelines [14] . Evaluation questions are criteria to evaluate the project and are guideposts towards fulfilling the process of change [15] . Identifying the theory of change is a key component of an evaluation plan, because it facilitates identifying the desired outcomes and evaluation questions to assess the project.

To strengthen our evaluation model, we also utilized community-based participatory research (CBPR) principles as described in Sandoval et al. 2011 [16] . We believed this to be an important inclusion, as Utah’s APCD development is merger of large-scale IT implementation and public health collaboration. The use of the MCE provides a framework for our evaluation plan. The use of the CBPR describes methods to carry out the collaboration; facilitating equity and positive group dynamics between the evaluation team and the project team and stakeholders.

We emphasized context, group dynamic processes, the APCD as an intervention, and outcomes in our work. Figure 2 provides a visualization of the collaborative evaluation model coupled with the core components of our evaluation process. In our work we considered the requirements for our evaluation including our stakeholders, relationships between organizations, funding available to undertake the evaluation, and the legislative mandate for the APDC, as well as

other important aspects of the context of the evaluation we were undertaking. We also used a process in which there was equity of stakeholder groups and participants. The concept of equity was critical to establish bi-directional communication essential to the collaborative evaluation approach. To iteratively develop the APCD, we focused on the improvements needed to produce required results such as creating data to assist providers in reducing the cost of healthcare, assisting the Utah Insurance Department with insurance rate review, and empowering consumers to make better decisions about healthcare expenses. Last, we focused on the desired outcomes in terms of being able to use the APCD to provide data to fulfill important uses cases determined by our stakeholders. Important uses cases include developing asthma measures, assessing falls in the elderly, and increasing price transparency for maternity episodes of care.

We developed the process of change statement as well as our evaluation plan, including our evaluation questions presented in Table 1, through collaborative feedback with project stakeholders in order to reach a unified goal. The theory of change provides what will happen if the project is successful. The evaluation questions help the evaluation team and the stakeholders determine what aspects of the project have been successful and have value. Our theory of change for Utah’s APCD was: “By improving the existing capability and functionality of the APCD, price transparency information will be provided to consumers, employers, researchers and the general public in Utah to support public health and health reform efforts”. Because the evaluation questions are critical, we also developed them iteratively with stakeholders. Stakeholders, such as employers, the public, insurers, and payers, all participate in healthcare decisions. Any or all are influential drivers in cost reduction [5] . Public health researchers at universities and the UDOH, and policy makers, are also interested in the quality of care and influential in optimal use of APCDs [3] . Gathering and engaging a stakeholder pool with diverse interests is important for development of the APCD [1] [6] . As described in other successful evaluation efforts, ongoing and systematic evaluation, including development and use of logic models and use of evaluation questions to guide the evaluation, are also important [2] . Based on the collaborative approach we used, the process of change, evaluation questions, logic models, and

other methods used in the evaluation were developed in partnership with the project team and key stakeholders.

We gathered feedback about the evaluation process with surveys. We used survey results to clarify the goals of the collaborative evaluation model by sharing the formalized evaluation plan including the process of change and evaluation process framework. We began with project leadership, subsequently met with the various stakeholder groups, and then provided updates on our progress throughout Year 2.

The three surveys focused on HIT, logic models, and collaborative evaluation (Tables 4-6). Our logic models were well received but there was confusion regarding the evaluation team’s role and questions about the utility of the collaborative evaluation model. Satisfaction with the model and understanding of its use were mixed. Half of the respondents found the process useful, one quarter were ambivalent and one quarter found the process intrusive. Satisfaction with the model and understanding of its use likewise varied: 50% found the model useful, 25% were ambivalent, and 25% found the model intrusive.

The majority of respondents specifically felt the evaluation plan positively contributed to the Cycle III grant project, while a minority felt neutral and an

even smaller minority felt that the plan did not positively contribute. This feedback indicated the need for ongoing communication about evaluation goals. Several respondents expressed that the logic model and collaborative model provide “valuable guidance” for “what tasks/activities need attention”. Others reported being unclear about the role of the evaluation team and did not appear to understand the actual focus of the evaluation. They felt the team should be “focusing on an evaluation of the impact of the project”, although discussions emphasized the focus on the plan’s processes.

Engagement with our collaborative stakeholders remained positive throughout Year 2, and we again prepared a survey to gather feedback. We designed three surveys in Year 2 to gather information from specific groups based on their work, increasing the potential for all respondents to provide valid responses. Survey questions ranged from use of the collaborative model and the evaluation team’s role in the project, to some of our more visible contributions, such as best practices tracking and logic models.

Feedback from our Year 2 survey yielded overwhelmingly positive results and showed that our stakeholders had increased confidence in our collaborative efforts. The majority of respondents specifically felt the evaluation plan positively contributed to the Cycle III project, while a minority felt neutral and an even smaller minority felt that the plan did not positively contribute. This was a noticeable improvement from the previous year’s survey, demonstrating improvement in sharing and utilizing the collaborative evaluation model in the APCD development project. We developed our evaluation plan through 71 communications with 16 project groups. Responses were overwhelming positive, with 90% of all responses falling under “Strongly Agree” or “Agree”. The remaining 10% were neutral, which all pertained to the use of best practices in SDLC. Respondents felt that the most valuable best practices were those in data quality (80% “Strongly Agree”; 20% “Agree”).

UDOH obtained funds from the Cycle III rate review grant to further develop the APCD [17] . Other states report using the APCD to evaluate specific use cases such as asthma and medical homes, and to understand evolving needs of the healthcare system [18] [19] [20] . Our collaborative evaluation framework was instrumental in the development and success of Utah’s APCD implementation to date. We found that communication is essential to effective collaboration. We promoted stakeholder engagement by extensive, ongoing contacts by email and meeting to explain the project, address concerns, and promote ownership of APCD among different participants. We recognize that we used a new applied practice in public health evaluation, and stakeholders are more accustomed to the outcome-based evaluation process. Although our model was initially viewed unfavorably by about 25% of respondents to our survey, we interpreted this response as stemming from the change in the evaluation process, timing of

which is new for our stakeholders.

Framework elements such as best practices are foundational to the support of use cases and are essential to accomplishing the project’s objectives. As noted in the APCD Development Manual, obtaining funding for further development is important. Collaborative program evaluation provided additional data, processes, and value to facilitate successful completion of the project. The evaluation process will benefit stakeholders by improving online pricing and cost transparency reports for consumers, employers, researchers, and the public in Utah.

The additional methods used in Utah may be beneficial for other states developing an APCD, especially if use cases of high value and HIT (such as analytic and other software) are being developed and used. The collaborative approach requires a significant number of meetings with stakeholders and this may not be feasible for all projects. Most important, best practices can play an important role in helping high risk, large scale HIT projects achieve success.

A collaborative program evaluation approach, including the use of best practices in developing and implementing enhancement for APCDs, builds on the foundation provided by the APCD Development Manual [1] . APCD development can be a dynamic process involving many constituent stakeholders. Because of the evolving nature of APCD development, the development team benefits from continual engagement of stakeholders and bi-directional feedback loops. Stakeholders found that the added use of best practices, logic models, and frequent feedback to practitioners facilitated the project’s success. Since the Collaborative Evaluation Model served a structural purpose, it was transparent to the project teams. Based on these results, when developing and improving APCDs teams should consider using logic models, a collaborative evaluation process, as well as best practices in security and privacy, large-scale HIT development, software development, and data quality.

The use of a collaborative approach in this APCD evaluation included key methods and tools, recommendations from the APCD Development Manual, the use of a collaborative evaluation model, logic models, and development and use of best practices as measures. Stakeholders felt that our transparent evaluation efforts facilitated the project’s success.

This publication is funded by CMS Grants to States to Support Health Insurance Rate Review and Increase Transparency in Health Care Pricing, Cycle III (Grant Number: 1 PRPPR140059-01-00), through a subcontract with Utah Department of Health Office of Health Care Statistics.

No conflict of interest is declared by the authors.

Waiver for human subject research by IRB # 00070963.

Garvin, J.H., Cardwell, J.H., Doing-Harris, K., Bolton, D., Snow, L.A., Hawley, C.W. and Xu, W. (2018) Collaborative Evaluation for the Utah All Payer Claims Database Capacity Enhancement. Technology and Investment, 9, 91-108. https://doi.org/10.4236/ti.2018.92007