We employed a mixed-methods approach, combining both qualitative and quantitative research methods, to gain a comprehensive understanding of the research problem. This approach allowed us to triangulate findings and increase the validity of our results and allowed us to delve deeper into the intricacies of healthcare big data processing. Qualitative methods, such as literature reviews and expert consultations, provided valuable contextual insights, while quantitative methods, like data analysis from the MIMIC-III Clinical Database, offered empirical evidence to support our findings.

The practical simulation built upon each method further reinforced the validity, reliability, and credibility of our results. By applying diverse approaches and cross-verifying our findings, we were able to mitigate potential biases and limitations inherent in any single research method. This triangulation of findings from various sources ensured a more balanced and nuanced understanding of the complex research problem at hand.

Ultimately, the mixed-methods approach emerged as the optimal strategy for our study. It enabled us to capture the multifaceted nature of healthcare big data processing and to derive more robust and insightful conclusions. By integrating qualitative and quantitative methods, we were able to delve deeply into the intricacies of the subject matter, providing a comprehensive analysis that enhances the relevance and applicability of our research findings. Details on the methods used, including containerization overhead, orchestration efficiency, and scalability, are provided in Tables 1 - 3 , respectively.

In addition to the literature review, we also consulted with industry experts and analyzed publicly available datasets to gain a deeper understanding of the practical challenges faced by healthcare organizations.

Through these methods, we identified key challenges and gaps in current ETL processes, including:

Summary of Challenges and Gaps

Our mixed-methods approach revealed a clear need for more efficient and scalable ETL processes in the healthcare sector. The challenges and gaps identified include resource inefficiency, limited scalability, and suboptimal ETL processes. These challenges and gaps served as the foundation for our research and experimentation, guiding our exploration of Containerized application, PySpark, Docker, as potential solutions.

Extract, Transform, Load (ETL) (Equation (1)) is a crucial process in data integration that involves extracting data from multiple sources, transforming it into a standardized format, and loading it into a target system for analysis and reporting [30] [31] .

The ETL process can be mathematically represented as:

ETL = (Extraction, Transformation, Loading) (1)

where:

Extraction = Data Collection + Data Cleansing + Data Mapping

Transformation = Data Transformation + Data Aggregation + Data Quality Check

Loading = Data Loading + Data Indexing + Data Retrieval

The ETL equation highlights the interconnectedness of the three stages, emphasizing that each stage is dependent on the previous one. A successful ETL process ensures that data is accurately extracted, transformed, and loaded, enabling organizations to make informed decisions based on reliable and consistent data [32] .

Our methodology is characterized by meticulous experimentation, emphasizing a sophisticated array of technologies:

This paper not only unearths key findings but also encapsulates pivotal insights with a pragmatic lens. Emphasizing the practical benefits of deploying PySpark within Docker containerized applications using Docker Compose. Our approach is tailored to streamline Big Data Engineering and ETL processes. As we navigate through this exploration, our focus remains unwaveringly committed to contributing insights that hold tangible value within the healthcare domain. The practical implementation is illustrated as shown in Figure 1 below.

Our experimental setup aimed to assess the efficiency and effectiveness of four distinct ETL processes within the healthcare domain, focusing on Big Data Engineering principles. We utilized the MIMIC-III Clinical Database, a widely used database in healthcare research, to simulate real-world data processing scenarios. Below, we detail our experimental setup, including the MIMIC-III database, configuration specifics, and the four ETL processes leveraging various technologies such as PySpark, Docker, Docker Compose, Python, Pandas, and PostgreSQL.

The MIMIC-III (Medical Information Mart for Intensive Care III) database is a comprehensive collection of de-identified health-related data from over 40,000 patients who stayed in critical care units of the Beth Israel Deaconess Medical Center between 2001 and 2012. It includes information such as demographics, vital signs, laboratory tests, medications, and more, making it a valuable resource for healthcare research and analytics [17] [18] .

The following tables provide a data dictionary for the sample datasets used in this research. The LABEVENTS table contains lab test results, while the PATIENTS table includes demographic details. Tables 4 - 5 are used to demonstrate data processing and analysis procedures in the research.

Our experimental environment was configured as follows:

1) ETL Process based on Python - Pandas: This process leveraged Python and the Pandas library for data extraction, transformation, and loading (ETL) tasks as shown in Figure 2 . Python’s versatility and Pandas’ powerful data manipulation capabilities were utilized to process the MIMIC-III dataset efficiently [42] .

2) ETL Process based on PySpark: PySpark, a Python API for Apache Spark, was utilized for this ETL process as shown in Figure 3 . Spark’s distributed computing framework enabled scalable and high-performance data processing, suitable for handling large-scale healthcare datasets like MIMIC-III [43] .

3) ETL Process based on Docker - Python/Docker Compose: Docker containers were employed to encapsulate Python-based ETL workflows as shown in Figure 4 . Docker Compose orchestrated the deployment of multiple containers, ensuring

seamless integration and scalability of the ETL process [44] [45] .

4) ETL Process based on Docker - PySpark/Docker Compose: This process combined the power of PySpark with the containerization benefits of Docker, as shown in Figure 5 . PySpark applications were containerized using Docker, and Docker Compose orchestrated the deployment for optimized scalability and reproducibility [43] .

To obtain the result data for each ETL process, we defined specific Python functions designed to fetch detailed performance metrics during each step of the workflow. These functions were deployed across all containers, tracking key parameters such as processing time, volume of data processed, and memory usage.

For each ETL process:

This structured approach enabled us to compile reliable performance data from the different environments, providing insights into the efficiency and scalability of each ETL method. The collected data formed the basis for the comparative analysis presented in the following sections.

In each ETL process, whether using Python-Pandas, PySpark, or Docker-based solutions, the data processing steps remained consistent across the different tools. Here’s a breakdown of the general workflow:

1) Data Ingestion:

The first step involved reading data from CSV files, specifically the LABEVENTS and PATIENTS datasets. These datasets contain patient information and lab results, which were critical for the analysis.

2) Data Merging:

The LABEVENTS dataset, containing lab test results, was merged with the PATIENTS dataset using the common SUBJECT_ID field. This step was crucial to link each patient’s lab test results to their demographic and health information.

3) Data Cleaning:

After merging, irrelevant columns that were not needed for the analysis were dropped to simplify the dataset and reduce complexity. This step ensures that only the most relevant data remains, improving processing speed and focus.

4) Data Transformation:

Date columns, such as birthdate and death date, were transformed into a standard datetime format, enabling easier manipulation and analysis.

These steps formed the backbone of the data preparation process, ensuring that the datasets were cleaned, merged, and ready for further analysis or reporting. The use of different tools, such as Pandas or PySpark, allowed the workflow to handle large volumes of data efficiently while maintaining consistency in the transformation logic across all implementations.

This section presents the experimental results obtained from two distinct ETL processes: one utilizing Python with the Pandas library, as shown in Figure 6 , and the other utilizing PySpark. The purpose is to compare their performance in handling data processing tasks.

The Python-Pandas ETL process exhibited moderate performance, with execution times ranging from 0 to 55 seconds as the dataset size increased up to 2 GB.

The PySpark-based ETL process demonstrated varying performance characteristics. For dataset sizes up to 0.25 GB, execution times ranged from 3 to 10 seconds. However, beyond this point, there was a noticeable drop in execution time, stabilizing around 8 seconds for larger datasets.

This graphical representation not only encapsulates the trends observed but also emphasizes the scalability and efficiency benefits inherent in utilizing PySpark for processing larger datasets compared to the Python-Pandas approach. Notably, the advantage of PySpark becomes pronounced for datasets exceeding 0.25 GB in size, making it the preferred choice for handling substantial volumes of data efficiently.

The summary of results presented in Table 4 aligns with the findings depicted in the linear graph. The Python-Pandas ETL process exhibits a linear increase in execution time proportionate to the dataset size, peaking at 55 seconds for a 2 GB dataset. Conversely, the PySpark ETL process showcases notably more efficient performance, characterized by execution times initially fluctuating but eventually stabilizing around 8 seconds for dataset sizes beyond 0.25 GB, as shown in Table 6 .

A scalable ETL process was successfully implemented using Python, containerized with Docker, and orchestrated with Docker Compose. The experiment involved deploying 10 containers, each representing a worker node, to process large datasets. The results, visualized in the accompanying graph, demonstrate the significant benefits of containerization and orchestration in enhancing the efficiency and scalability of the ETL process. The graph, Figure 7 , showcases the processing time for each worker node, illustrating the improved performance and distributed processing capabilities achieved through containerization and orchestration.

Containerization Overhead: Impact of Docker containers on overall performance.

Orchestration Efficiency: Effectiveness of Docker Compose in managing multi-container applications.

Reproducibility: Consistency and reliability of results across different environments.

The Dockerized Python ETL process exhibited slightly increased execution times due to containerization overhead.

Docker Compose effectively managed the orchestration of multiple containers, ensuring streamlined deployment and management.

Results were highly reproducible across different environments, indicating the reliability and consistency of the Docker-based approach.

The bar chart illustrates the performance of each container within the Dockerized Python ETL process across different dataset sizes (ranging from 0.25 GB to 2 GB).

The median range of execution times for each container varies from 0.2 seconds to 12 seconds, indicating the impact of containerization overhead and orchestration efficiency.

Despite the slight increase in execution times due to containerization, Docker Compose effectively manages the orchestration of multiple containers, ensuring streamlined deployment and management of the ETL process.

The consistent performance across different environments highlights the reproducibility and reliability of the Docker-based approach for deploying ETL processes.

This section presents the experimental results of a containerized ETL process, leveraging Docker for PySpark application deployment and Docker Compose for seamless orchestration. The outcomes are based on a scaled deployment of 4 containers, offering valuable insights into the performance, efficiency, and scalability of the containerized ETL process. The results demonstrate the benefits of containerization, including improved resource utilization, enhanced productivity, and streamlined workflow management. The performance gains are shown in the following graph, Figure 8 , which illustrates the significant improvements in processing time and throughput achieved through containerization and orchestration.

Performance Optimization: Impact of containerization on PySpark performance.

Scalability Enhancement: Utilization of Docker Compose for optimizing scalability.

Environment Consistency: Reproducibility of results across various deployment environments.

The Dockerized PySpark ETL process demonstrated efficient performance, leveraging Docker’s containerization benefits without significant overhead.

Docker Compose effectively managed the scaling of PySpark containers, optimizing resource utilization and improving performance.

Results were consistent and reproducible across different deployment environments, indicating the robustness and reliability of the Docker-based approach.

Performance Comparison Bar Chart.

The bar chart illustrates the performance of each PySpark container within the Dockerized ETL process across different dataset sizes.

With only 4 containers, the Dockerized PySpark ETL process showcases efficient performance, demonstrating the scalability and optimization benefits of Docker Compose.

The consistent performance and reproducibility across different deployment environments highlight the reliability and robustness of the Docker-based approach for deploying PySpark-based ETL processes.

The following tables provide a summary of the results from the experiment, which was discussed in earlier Figures 6-8. The results are presented in four tables, each focusing on a specific aspect of the containerized ETL process.

This table presents the metrics related to containerization and orchestration, including containerization overhead, orchestration efficiency, and reproducibility, as shown in Table 7 . The results show the benefits of using Docker and Docker Compose for containerization and orchestration.

This table compares the scalability, efficiency, and reproducibility of traditional Python-Pandas, PySpark, and containerized Python and PySpark ETL processes. The results demonstrate the advantages of containerization in improving scalability, efficiency, and reproducibility, see Table 8 .

As shown as Table 9 , this table shows the execution time in seconds for different dataset sizes using traditional Python-Pandas, PySpark, and containerized Python and PySpark ETL processes. The results illustrate the improved performance of containerized ETL processes, especially for larger datasets.

As shown as Table 10 , this table highlights the benefits of using containerization and orchestration with Docker and Docker Compose, including improved scalability, increased efficiency, enhanced reproducibility, simplified deployment, and streamlined management. The results demonstrate the advantages of using containerization and orchestration in ETL processes.