Abstract

In 2023, the challenge of achieving COVID-19 vaccination targets in Nigeria, especially at the subnational levels, was enormous. The Technical Assistance for COVID-19 Vaccine Deployment Project was implemented in 18 states to strengthen coordination, data use, and institutional capacity. A Most Significant Change evaluation was conducted to identify outcomes attributable to the TA intervention. Six states, one from each geopolitical zone, were purposively selected. Fifty-five stakeholders participated, and 32 significant change stories were randomly selected and analyzed. Findings revealed notable improvements across all domains. COVID-19 Technical Working Groups demonstrated stronger planning, coordination, and functionality, and increased use of data for planning, resource allocation, and targeting. Institutional strengthening was reflected in greater government ownership, enhanced supervision, and improved capacity of state actors to manage data and logistics. The evaluation highlights the critical role of targeted technical assistance in strengthening health systems in resource-constrained settings and addressing key response gaps.

Keywords

COVID-19, Technical Working Group, Vaccination, Technical Assistance, Planning and Coordination, Data Use, Institutional Strengthening

Share and Cite:

1. Introduction

As of 11 March 2020, there was confirmation of the worldwide spread of the disease as a pandemic by the World Health Organization, with 118,000 cases and 4291 deaths reported across 114 countries (NCDC, 2023). The first case of COVID-19 in Nigeria was confirmed on 27 February 2020. The case was a 44-year-old Italian citizen who arrived in Nigeria through the Murtala Mohammed International Airport, Lagos, on a flight via Milan, Italy (NCDC, 2020; WHO, 2020a).

The continuous emergence of new COVID-19 strains means the global pandemic remains a persistent concern for public health. Addressing this challenge, the development and deployment of the COVID-19 vaccine have emerged as crucial strategies. Presently, the COVID-19 vaccine stands as the most effective method to exert control over the pandemic (Adigwe & Onavbavba, 2024). In March 2021, following the implementation of various non-pharmaceutical interventions with varying degrees of effectiveness, Nigeria initiated its COVID-19 vaccination program. At this point, the nation had already reported 162,593 confirmed cases, with Lagos and the Federal Capital Territory (FCT) bearing the largest burden of the disease. Aligned with the global goal of eradication, Nigeria set an ambitious target of vaccinating 40% of its population by the end of 2021, to increase the complete vaccination rate to 70% by December 2022. This underscores the nation’s commitment to mitigating the impact of the virus and contributing to the international effort to achieve widespread immunity.

In response to the persistent global health challenge posed by COVID-19, the National Primary Health Care Development Agency (NPHCDA) in Nigeria has taken the lead in orchestrating a comprehensive vaccination campaign, collaborating closely with state agencies and boards. This initiative has entailed the establishment of the COVID-19 crisis communication center, the implementation of various campaign models to intensify service delivery, and targeted measures to fortify logistics management, enhance accountability, and provide supportive supervision. Additionally, the adoption of the Electronic Management of Immunization Data (EMID) system has been a notable advancement.

Despite these concerted efforts, Nigeria fell short of the global vaccination target, achieving only a 54% vaccination rate by the conclusion of 2022 (Ogunniyi et al., 2023). This underscores the challenges faced by the nation in achieving widespread immunization coverage and highlights the need for continued strategic interventions to bolster vaccination efforts and curb the impact of the ongoing health crisis (Olu-Abiodun et al., 2022).

The challenges contributing to lower vaccination rates in Nigeria span both macro and micro levels. On a macro level, there is evident suboptimal management of COVID-19 vaccination programs, particularly at the subnational levels. This is compounded by the verticalization of the COVID-19 vaccination programs, limiting their effectiveness. The financial commitment from the government has been insufficient, impeding the seamless execution of vaccination initiatives. Regulatory hurdles and the inability to fulfill vaccine storage requirements pose additional obstacles, further complicating the vaccination landscape (Ayenigbara et al., 2021). Additionally, the limited shelf-life of COVID-19 vaccines presents a time-sensitive challenge, requiring meticulous planning and swift execution to maximize vaccine utilization within the specified timeframe. Addressing these macro-level issues is pivotal to overcoming the barriers hindering the achievement of higher vaccination rates in the country.

At the micro level, the challenges contributing to lower vaccination rates in Nigeria are multifaceted. Hesitancy and a low acceptance rate among the population play a significant role, reflecting concerns and uncertainties regarding the COVID-19 vaccines. Distrust in government actions and policies further hampers vaccination efforts, undermining public confidence in the vaccination programs. Limited access to vaccines poses a critical challenge, hindering the timely and equitable distribution of immunization resources. The inability to reach vulnerable communities promptly, exacerbated by issues of insecurity and displacements, further complicates the successful implementation of vaccination campaigns (Ogunniyi et al., 2023).

The use of different vaccines has generated complexities, as concerns and misinformation around specific vaccines contribute to hesitancy. Poor communication strategies, coupled with weak cold chain systems and logistics management capacities, impede the efficient delivery of vaccines. Inadequate infrastructure exacerbates these challenges, hindering the seamless execution of vaccination programs. The perennial shortages and maldistribution of human resources for health are persistent issues, particularly impacting rural areas and depriving them of access to quality health services (Ayenigbara et al., 2021).

It is essential to recognize that the scale and scope of these challenges vary across states and between rural and urban settings, necessitating targeted and context-specific interventions to address the unique dynamics at play in different regions.

2. Project Context

To address the challenges faced by the COVID-19 vaccination program, a crucial partnership between GAVI, the Nigerian government, and eHealth Africa was formed. eHealth Africa (eHA) is a non-governmental organization focused on improving health systems, with core technical expertise to build stronger health systems through the design and implementation of data-driven solutions that respond to local needs and provide underserved communities with tools to lead healthier lives. The partnership for the project with the Nigerian government was primarily driven by the National Primary Health Care Development Agency (NPHCDA), the States Primary Health Care Development Agencies/Boards (SPHCDA/Bs), the State Emergency Routine Immunization Coordination Centre (SERICC), the State Emergency Maternal and Child Health Intervention Centre (SEMCHIC), and Coordination Platforms. This collaboration, known as the Technical Assistance for COVID-19 Vaccine Delivery Project (TA-CVDP), operated across 18 states and aims to enhance planning, coordination, and vaccination strategy.

The project is intended to optimize the performance of the coordinating structures by supporting the Government and its partners to ensure optimal management of COVID-19 vaccination programs. This is achieved through the integration and leveraging of existing platforms like the Emergency Operations Centre (EOC) and the Nigeria Governors’ Forum (NGF) to support the COVID-19 vaccination process. The 18 states selected cut across the 6 geopolitical zones and were the states with the lowest vaccination coverage as of February 26^th, 2023.

Setting up a structure for optimal management of the COVID-19 vaccination program across the 18 states was intended to strengthen the coordination of the stakeholders and activities in the program and enable the government to effectively implement the vaccination program and achieve the 70% target of Nigerians vaccinated against COVID-19.

The technical assistance project through eHealth Africa supported Planning & Coordination through the reactivation and strengthening of Technical Working Groups (TWGs), regular coordination meetings (weekly/biweekly/monthly), deployment of action plan trackers, support for Emergency Operations Centres (EOCs) and coordination platforms, and the development and implementation of performance optimization plans. Data Use was supported through capacity building on EMID and data management systems, data triangulation (Electronic Management of Immunization Data, call-in data, migration of population data), support for real-time data tracking, validation, and feedback loops, and strengthening of state-level data analysis and reporting. Institutional Strengthening was achieved through capacity building across pillars (data, logistics, planning), supportive supervision systems strengthening, financial and governance system improvements (e.g., accounting tools), and support for the routinization of COVID-19 vaccination. These three coordination domains were delivered through various channels, which include: in-person Technical Working Group (TWG) and coordination meetings, on-the-job mentoring and technical accompaniment, trainings and workshops (state and LGA levels), deployment of tools (trackers, reporting tools, EMID support), and support via coordination platforms (EOC; State Emergency Routine Immunization Coordination Centre, SERICC; State Emergency Maternal and Child Health Intervention Centre, SEMCHIC).

The technical assistance project was implemented between February and December 2023 across 18 states, with continuous engagement of stakeholders, with eHealth Africa supporting over 500 TWG meetings, routine supervision, and real-time campaign, iterative capacity building of stakeholders, and feedback cycles of public health events.

Key stakeholders supported were the State & LGA Health Teams (SPHCDA, M&E, RI officers) who were supported in data analysis training, planning support, and supervision. Members of the TWG & Coordination Platforms were supported through meeting facilitation, tools (trackers), and funding/logistics. Health Facility Workers (HFW) were supported through training on EMID, vaccination reporting, and routinization, while Government Institutions were provided with institutional capacity strengthening, planning tools, and financial management systems. The MSCE was designed to ascertain the most significant change that has occurred, unravel the mechanisms through which it transpired, and delineate the pivotal role the TA support played in bringing about any change.

3. Methodology

The evaluation was conducted by independent consultants not affiliated with eHealth Africa using a Most Significant Change Evaluation (MSCE) approach. The MSCE was most suited for this evaluation because of the intricate nature of Nigeria’s COVID-19 vaccination program, which involves complex, diverse, and interlinked systems, contexts, and multiple actors working towards achieving the same goal of achieving Nigeria’s COVID-19 aspirations. To gather insights, a secondary content analysis of the significant change stories was employed as a summative assessment of the Technical Assistance for COVID-19 Vaccine Delivery Project (TA-CVDP). This method, outlined by Rick and Davies, has been detailed elsewhere and was deemed apt for this evaluation due to its suitability in capturing the complex, divergent, and emergent outcomes observed within diverse systemic and geographic contexts throughout the project (Rick, 2005).

3.1. Selection of State

The evaluation used a purposive multi-stage sampling approach, as Nigeria is divided into 6 geopolitical zones, each zone comprising an unequal number of states. The three domains of interventions were implemented in 18 states, with three states in each geopolitical zone. In each zone, a participatory approach was used to purposely select 1 state (Table 1) based on the following criteria: accessibility, approval from the state stakeholders, and the magnitude of the TA intervention.

Table 1. Domains of interest in the six evaluated states.

3.2. Selection of Respondents

The Evaluation Team planned to interview 66 stakeholders representing the coordinating structures responsible for COVID-19 vaccination implementation in each selected state. Through a collaborative process with state stakeholders, the relevant participants for the MSCE from the thematic areas were assessed at the point of project entry to the states. These participants were state government staff and partners who were actively involved in direct intervention efforts to support project states’ improvement in COVID-19 Vaccine Delivery to achieve the expected national coverage. This purposeful selection ensured a balanced representation of the beneficiaries of the project’s implementation activities. The proposed respondents were categorized by the domain most relevant to them. In addition, the stakeholders selected were domain-specific. That is, some stakeholders could only respond to the Data Use interventions, while others would only respond to Coordination and Planning, and notably, we selected some stakeholders who could respond to the 3 areas of intervention. However, the Evaluation Team was able to interview 55 of the proposed stakeholders, which gave rise to a response rate of 83%. Each interview generated a “Most Significant Change (MSC) story”; 55 total stories were collected, but 32 stories were selected for narrative analysis. The selected stories were based on completeness and domain coverage, not purely randomly, and the remaining 23 stories were used for a docuseries. This ensures clarity in interviews to stories (1:1 relationship) and the final analysis prioritizes rich, complete, and domain-representative stories.

3.3. Data Collection Tool and Process

Due to the pros and cons of qualitative data collection methods, the COVID-19 Vaccine Deployment Project Evaluation used audio recorders and/or Microsoft Dictate, as it allowed for capturing respondents’ emotions and attitudes towards the project, and in the case of Microsoft Dictate, it generated the raw transcription notes, which were later cleaned and formatted for analysis. Most importantly, the interviews were facilitated with a question guide.

Given that respondents were largely stakeholders directly involved in the implementation of the Technical Assistance for COVID-19 Vaccine Deployment Project, including government actors and partners working closely with eHealth Africa, there is potential for social desirability bias, where participants may overemphasize positive outcomes or attribute success to the project due to perceived affiliations or expectations. Additionally, the involvement of eHealth Africa in identifying participants could have influenced respondents’ willingness to provide critical feedback.

To mitigate these risks, several steps were taken. Data collection was conducted by experienced and, in part, independent interviewers, including an external consultant responsible for qualitative analysis, which helped reduce interviewer bias and power dynamics. Respondents were engaged using structured interview guides, and interviews were recorded to ensure the accurate capture of perspectives. Furthermore, confidentiality assurances and the use of thematic analysis with triangulation across multiple respondents and locations strengthened the credibility of the findings and reduced the likelihood of biased interpretation.

3.4. Data Processing and Management

The qualitative data/stories collected through guided interviews were analyzed using a thematic content analysis approach. This involved transcribing the interviews and reading and re-reading the transcripts to identify recurring themes from the stories of change.

A hybrid coding approach was employed by applying a set of a priori codes to the transcribed text (deductively) and adding newly emerging codes (inductively) as the analysis progressed. We generated 135 codes applied to 810 statements along 22 themes, and the findings were synthesized to draw overarching insights and deductions.

Table 2 shows the category of respondents and the socio-demographic profile of participants. Through the thematic analysis process, key patterns were identified, and insights were obtained from the most significant change stories collected from the beneficiaries of the COVID-19 Vaccine Deployment Project, as well as the project’s impact on planning, coordination, data utilization, and institutional strengthening across the selected states. The thematic data analysis was carried out using NVivo software version 14.

Of the 55 stories collated, 23 stories were analyzed for the docu-series production, while 32 stories were used to produce the narrative report. The 32 stories were selected based on data (story) completeness across the domains for transcription and analysis. This was done by randomly selecting 5-6 stories on Data Use, Planning & Coordination, alongside a stakeholder story that captures the 3 domains of intervention. This report focuses on the findings from the 32 stories gathered in the narrative report.

Table 2. Distribution of participants by the domains of interest in six evaluation states

3.5. Ethical Considerations

The study received ethical approval from the National Health Research Ethics Committee of Nigeria (NHREC/01/01/2007-20/02/2025). This study was conducted in accordance with the ethical standards of the Helsinki Declaration. Before they participated in the study, the study participants were informed about the study’s objective, how voluntary their participation is, and the opportunity to withdraw at any time without any consequences. Research Assistants also thoroughly explained the consent form and addressed any questions or concerns from the participants. Following this, oral consent was obtained from the participants and documented for the record.

3.6. MSC Evaluation Limitation

By nature, the most significant change methodology is not designed to provide comprehensive information about the changes brought about through a project or program. It is not designed to address typical change, but rather the most significant change. This is not just a weakness of the technique itself, but is inherent in purposeful sampling, selecting the most information-rich stories to analyze. Though this evaluation attempted to ensure a spread of respondents across the geopolitical zones and domains of assessment, interpretation of findings was done cautiously with the hindsight that this is a summative evaluation of the COVID-19 Vaccine Delivery Project within its peculiar implementation contexts in the selected six states and is not generalizable to other contexts without supplementary evidence. However, this study avoided interpretation biases and strengthened the internal validity of findings through independent analysis of the data by an external consultant and the triangulation of codes and emerging themes across respondents and locations.

4. Results

4.1. Participants’ Socio-Demographics

According to Table 3, the stakeholders’ stories used for the report were five from each state except FCT and Oyo, which both had six. Eleven (34%) of the participants’ stories provided perspective on all three domains of assessment, while other participants responded on the individual domains: data use, 6 (19%); institutional strengthening, 7 (22%); planning and coordination, 8 (25%). There were more male story participants, 22 (69%), than female, 10 (31%).

4.2. Notable Changes Attributed to the Technical Assistance Intervention

All 32 participants shared their perspectives on the changes they perceive have been contributed to by the project across the three domains of assessment-planning and coordination; data use for COVID-19 vaccination and actions; and institutional strengthening. The following sections describe the most significant changes and highlight the other changes reported by respondents per domain,

Table 3. The socio-demographic profile of the participants.

including the reasons stated by the respondents for considering the changes important and the interventions or actions of the TA support that contributed to the changes.

4.2.1. Improved Planning and Coordination of the COVID-19 TWG

Before the commencement of the support of TA-CVDP to the states, the following were the perceived gaps reported by the assessment participants regarding the planning and coordination of the COVID-19 vaccination: fewer partners supporting COVID-19 vaccination, irregular TWG meetings, inadequate TWG proceedings and action plans, low participation of relevant stakeholders, and low motivation of state teams to contribute to COVID-19 response activities. After the TA Support interventions, the findings from this study showed that the narratives have changed. Respondents broadly identified five distinct significant changes related to planning and coordination. As presented in Table 4, the most popular significant changes (identified by 10 out of 17 respondents for the domain) were the reactivation or increased functionality of the TWGs and other structures, and the increased participation of relevant stakeholders in the coordination platforms.

Table 4. Reported changes in planning and coordination of COVID-19 TWG across the six evaluations

Below are the descriptions of these most significant changes, the reasons why they are considered most important, and the contribution of the TA intervention.

1) Enhanced coordination of interventions and teams: The COVID-19 Vaccine Deployment Technical Assistance (TA) project was reported to have significantly contributed to enhanced coordination of interventions and teams. Respondents from Borno and Katsina states attributed the improved coordination to the comprehensive support received from the TA intervention, both technically and otherwise, through the project.

A respondent from Borno state emphasized the invaluable contribution of the TA intervention, stating, “Without the TA support, especially both technical and otherwise, I don’t think it would be feasible to gather all these organizations, structures, and stakeholders, and roll out all these interventions, gather data from all those interventions, a lot of things. The contribution has been valuable, and we look forward to more collaboration.”

Similarly, a respondent from Katsina state highlighted how the project facilitated the integration of COVID-19 issues into other structures, stating, “We have integrated all our series of meetings with COVID-19. So, she [TA personnel] helped integrate the State Emergency Routine Immunization Committee (SERIC) and other services with COVID-19. With their [TAs] coming, I think the project has encouraged all the teams to work together, integrating all activities, SERIC, State Emergency Maternal and Child Health Intervention Centre (SEMCHIC), and other Emergency Operation Centre (EOC) activities with COVID-19.” This affirms the positive impact of the project in fostering collaboration and synergy among various healthcare initiatives.

2) Better tracking of TWG action plans (Action Plan Tracker): The insufficient documentation of TWG proceedings and the lack of proper follow-through on agreed action points that were experienced before the TA support across all states has shown a significant and consistent change, as noted across all six states, in that there is improved tracking of Technical Working Group (TWG) action plans through the action-tracker initiative. Respondents from all states acknowledged the valuable contribution of the action plan tracker developed by the Technical Assistant Team, emphasizing its significance in proper planning and coordination of COVID-19 vaccine deployment.

In the FCT, a cross-cutting respondent highlighted the transformative impact, stating, “It was a continuous action tracker developed by eHA (the TA team), so you were able to see what we have done, what we have not done. When we have meetings, we’re able to look at all the reds and say no, go back up, we’re not done with this one; we need to sort this out and all that. So, it made our meetings better.” This underscores how the action plan tracker has not only improved the tracking of activities but has also become an integral management tool for leaders, preserving institutional memory and promoting shared accountability among stakeholders involved in COVID-19 vaccination efforts in the state.

3) Routinization of evening debrief meetings during immunization campaigns. The Planning and Coordination respondent from Akwa Ibom state identified the routine of evening debrief meetings during immunization campaigns as the most significant change. This notable shift was attributed to the technical support provided by the TA support in convening and facilitating these meetings. The respondent elaborated on how this routine has significantly contributed to the ability to respond to challenges in real time, thereby enhancing the overall effectiveness of immunization campaigns.

In the respondent’s words: “The most significant change has been the evening review meetings. These have enabled us to respond to challenges in real time, not necessarily having to wait until the end of the month to see those challenges arise. So, since we have these meetings very regularly, we could have a shortened response time, as short as two days or even one day. And we are able to address challenges and achieve more. So, I think that, for me, has been a very good improvement.” This underscores the value of regular debrief meetings in addressing challenges promptly and maximizing the effectiveness of immunization campaigns.

4) Reactivation or Increased Functionality of TWGs and Other Structures: A commonly cited and significant change across five states is the reactivation or increased functionality of Technical Working Groups (TWGs) and other structures, driven by the technical assistance support of the COVID-19 Vaccine Deployment Project. Respondents emphasized the importance of more regular coordination meetings facilitated by this support. These meetings play a crucial role in ensuring proper planning, fostering synergy among various actors, and facilitating regular reviews of progress to address emerging challenges.

“Coordination is vital because anything that you do, if you don’t have coordination, you begin to fail because that is where you need to sit down and discuss issues so that you can sort it out.” ...Cross-cutting respondent, Katsina State

An Akwa-Ibom respondent succinctly summarized this change, stating, “Yes, I can say the most significant change from the planning and coordination perspective is the frequent meetings, coordinating, ensuring that everybody is doing what he is supposed to do, and ensuring accountability is in place.”

The respondent further described how the TA support contributed to the improved functionality of the TWG by actively pushing for the meetings to happen. Similarly, a cross-cutting respondent from Oyo state attributed the reactivation of the TWG to TA’s support, stating, “When the project started, they were able to help us to resuscitate it.” Additionally, an institutional strengthening respondent from Oyo state explained how providing funding support for the meetings played a role in the improved functioning of the TWG, stating, “The TA support came in to support the meeting and provided some refreshments, which I think helped.”

“I can’t recall a meeting being held in the last 12 months before TA support came on board.” ...Coordination participant, Katsina

“Now somebody is pushing, ensuring things are done the way they are supposed to be. At least even the people from the state will know that this is our mandate, and we have somebody who is pushing us to ensure that this thing has been done.” …Coordination participant, Katsina

It was gathered in this study that the issue of irregular meetings within the vaccination coordination structures was reported by most participants, with consistent observations from Katsina, Akwa-Ibom, and Ebonyi states. The irregularity in meeting schedules was attributed to a lack of funding availability. Participants expressed that the withdrawal of partners who previously provided funding for these meetings as part of COVID-19 interventions had adversely impacted the coordination structures’ ability to convene regularly.

5) Increased participation in coordination platforms: Before the TA Support project, the TWGs experienced sparse attendance in the limited number of meetings conducted before 2022. According to a respondent from Borno, “Before the arrival of the partners, we were not experiencing turnouts; the individuals expected to attend the meetings in 2022 were frequently absent.” This reveals the underlying challenge of garnering active involvement from key stakeholders, emphasizing the importance of strategies to enhance engagement and participation in future vaccination initiatives.

The MSCE findings show that the TA support project contributed to the increased participation of critical stakeholders in coordination platforms. This was particularly attributed to the technical and logistical support provided by the Technical Assistance (TA) Project. Respondents highlighted that this increase in participation was facilitated by automated advanced calendar invites and reminders, as well as refreshments provided through the TA support intervention.

A respondent from Ebonyi succinctly expressed this change, stating, “Stakeholders that were supposed to be driving the meeting, most times they don’t take it seriously. They are not always there, but I knew that when eHA started supporting that meeting, it attracted a lot of people that are supposed to be there.” This change was considered significant as it eased the process of obtaining government buy-in for addressing challenges, enriched conversations, brought forth more ideas, and enhanced the utilization of data to drive decisions.

According to a cross-cutting respondent from Ebonyi state, “Now we have more people coming, more contributions being made, and more ideas being generated. And then for data, we now deal a lot with data for decision-making. That is a striking one for me when it comes to data. So, SERICC, the state at large, does not just make decisions abstractly, but now we make data-informed decisions.”

Respondents from the Federal Capital Territory (FCT) highlighted the enhanced partnership and collaboration with various actors within the Technical Working Group (TWG) to address gaps in COVID-19 vaccine deployment. The respondent emphasized the complementary nature of eHealth Africa’s TA support with the efforts of other stakeholders. Clear mention was made of collaboration with organizations such as WHO, UNICEF, and Government MDAs, contributing to improved planning and coordination of COVID-19 vaccine deployment.

Through this study, it was found that the TA intervention has also been perceived to improve COVID-19 vaccination coverage among people who received at least one dose of the vaccine through performance optimization plans. Respondents from the FCT and Borno highlighted how the operationalization of these plans has enhanced the effectiveness of COVID-19 vaccine deployment processes. According to a SERIC leader in Borno State, “A lot of key activities were being planned to improve the uptake of this COVID-19. And I can assure you that we have seen the improvement in COVID-19 vaccine uptake.” Another respondent from Akwa-Ibom emphasized the connection between the activities of the technical working group and the promotion of efficiency in vaccine deployment through coordination and joint reviews.

4.2.2. Enhanced Data Use for COVID-19 Vaccination Planning and Action

Most of the respondents referred to some data gaps that existed around data use for COVID-19 vaccination planning and action. The data issues identified included data transmission and flow issues from the health facility to the LGA to the states. At each stage of data use, it was found that significant gaps exist around data completeness at each stage of transmitting vaccination data from the community to the LGA and state levels. This has resulted in substantial backlogs of uploaded data on the designated platform. Another critical data gap was the low capability for data analytics and use. Most respondents generously admitted to inadequate capacity at the state level to analyze the COVID-19 vaccination data and generate any meaningful insight that could be used to guide planning and actions. At the time of this study, the states relied on the analysis carried out at the national level, which naturally lacked the granularity that could have been more useful for informing local actions at the state level. After the introduction of TA support that involved building the capacity of state stakeholders on data use, the changes highlighted in Table 5 were echoed by the respondents across the states evaluated.

Nineteen (19) respondents identified three main significant changes in data use for planning and action.

Table 5. State distribution and relative frequency of each significant change in the six study states.

1) Timely data availability for performance review and problem-solving: The most significant change, identified by nine respondents, revolves around the timely availability of data for performance review and problem-solving. This improvement encompasses the prompt submission of data from the sub-national level to the state, and respondents attribute this positive shift to the capacity building and accompaniment provided by eHealth Africa (eHA). This has significantly empowered field and Local Government Area (LGA) teams to address discrepancies between the Electronic Management of Immunization Data (EMID) and call-in data. It has also enhanced their capacity to troubleshoot and report on designated platforms.

2) Increased data use for planning, targeting, and other actions: Increased data use for planning, targeting, and various actions emerged as a unanimous and highly impactful change acknowledged by respondents across all six states, attributed to the COVID-19 Technical Assistance (TA) project. Notable among the actions informed by data is the triangulation of data from different sources to plan vaccination coverage. Respondents elaborated on how migration data and routine vaccination data have been combined with the project’s “walk-through microplanning” to enhance the accuracy of target population estimation. This, in turn, has provided a more realistic approach to monitoring progress.

Yes, people can now effectively use data for action. My staff can come to me and say, based on this in this area; this is what we analyzed. This is what I think we need to build as a form of intervention. We can monitor our data in real time, to even be able to monitor progress and see where we are doing very well, where we are lagging, and what can be done. And you can only do this when interrogating your data and juxtaposing it... Cross-cutting Respondent, Oyo State

Some respondents emphasized the use of data to allocate human and other resources according to the right target population as the most significant change brought about by the COVID-19 TA project between 2022 and 2023. Others highlighted the tracking of field team efficiency, providing prompt feedback for corrective actions, and utilizing data to promote stewardship and accountability for available vaccines in the state.

According to the respondents, the use of data has played a pivotal role in enabling states to make informed decisions. A cross-cutting respondent from Borno succinctly summarized this impact, stating, “Without data, it’s very difficult to decide. Even if we make decisions, it may be a wrong decision. However, once we had the TA Support intervention for the data department, we were able to see clearly where we were having problems. And that allowed us to strategize.” This strengthens the transformative role of data in guiding strategic decisions and fostering effective planning and implementation of vaccination initiatives.

3) Improved capacity of state teams to lead data analysis: A most significant change in the realm of data use, as identified by a cross-cutting respondent from the Federal Capital Territory (FCT), is the improved capacity of state teams to lead data analysis. This perspective is in harmony with the sentiment of a cross-cutting respondent from Katsina, who emphasized a notable enhancement in the capability of state teams to “carry out data analysis, share feedback, and use even the data”. According to these respondents, this change is crucial for fostering ownership and sustainability in data-related activities.

In addition to the above significant changes, respondents mentioned additional outcomes attributed to the COVID-19 TA project. These include:

4) Improved quality of data: Respondents highlighted a notable improvement in the quality of data in 2023 compared to 2022. They emphasized that enhanced data quality has had a direct impact on the overall quality of decisions made. A respondent from Oyo State succinctly expressed this connection, stating, “Our data has improved, even the way we make our decisions now has improved.”

5) Improved leadership support for data availability: Respondents from Oyo, Borno, and the Federal Capital Territory (FCT) confirmed an increase in leadership support and commitment to data availability (Table 5). Instances were recounted where TWG leadership issued warnings to non-responsive Local Government Area (LGA) and health facility staff. Similar leadership interventions were reported concerning underperforming LGA teams in terms of vaccine coverage performance. According to the respondents, these interventions have significantly boosted the availability of data for decision-making.

A cross-cutting respondent from Borno state highlighted this aspect, stating, “One of the strengths as well is the prompt tracking of this LGA M&E and those taking leadership because we have to come all out and take ownership. All the DPRS, the IM, and every senior stakeholder must come out to say no, we cannot continue this nonchalant attitude. How can a field worker go out to vaccinate a lot of people for M&E just to collect and send it to the State for decision-making, and not be able to do that? So, we started taking stringent measures, and that has yielded positive results.”

Respondents who identified prompt data availability as a significant change recognized it as a foundational requirement for other actions in the data value chain. They emphasized that if prompt data availability is negatively impacted, data use will be suboptimal. As articulated by a data use respondent from Katsina, “If data has not been reported, what will you analyze? You will have nothing to analyze and then to use.” This underscores the critical role of leadership support in ensuring timely and effective data availability for informed decision-making.

Overall, it was reported that the COVID-19 TA project has significantly impacted data use, particularly in achieving a more accurate measurement of the actual target population and addressing challenges associated with inadequate, obsolete, and unreliable demographic data in Nigeria. This issue is further compounded by insecurity in the North-east and North-west regions, leading to frequent displacements. The state target population used by the NPHCDA to determine the vaccination coverage for the state was high, and based on insecurity that led to migration, the target population of some settlements and, invariably, the LGA, was unrealistic. Therefore, there was a call to work with a realistic target that reflects the actual target population. Technical support was provided to the Monitoring & Evaluation (M&E) working group, which engaged with the LGA M&Es to develop an actual target population by leveraging International Organization of Migration (IOM) data at the LGAs and projected census data. The new state target population was then used to measure the vaccination coverage for COVID-19.

The project’s initiative to triangulate projected population figures with other available sources, such as IOM data on internal displacements, and validating through walk-through micro planning has not only promoted the use of data for planning and progress monitoring, but has also increased the subjective quality of the target population data. This subjective quality has been identified as a stronger determinant of continuous data use than objective data quality.

4.2.3. Institutional Strengthening

After the well-acknowledged technical assistance on the COVID-19 Vaccination deployment project, eighteen (18) respondents for this domain identified three most significant changes—the most popular of which is increased government ownership and commitment to COVID-19 vaccine deployment (Table 6).

“The most significant change for me is having the trained or built capacity of the local government Cold Chain officers.” ... Cross-cutting Respondent, Ebonyi State

1) Improved pillar-based capacity: The described changes pertain to data and reporting, vaccine logistics, planning, and coordination at both the state and Local

Table 6. Distribution of reported most significant changes related to institutional strengthening in the six studied states.

Government Area (LGA) levels. Respondents alluded to the observable increase in the capacity of staff in these thematic areas or pillars. A respondent from Borno State expressed this sentiment, stating, “Some of the support, especially the data control room, the capacity building of those fragile pillars among the departments and the pillars that we have, especially the Technical Working Group (TWG).” Another respondent confirmed the outcomes of these capacity-building efforts, stating, “So, these are some of the turnarounds actually because we quickly started seeing the impact of those institutional capacities.”

Another respondent from Borno state hinted that these capacity-building initiatives were tailored to address the gaps identified in a prior assessment conducted during the life of the project. He traced the improved capacity to the technical assistance intervention, stating, “Yeah, with the support received, I know there is a lot of contribution from eHealth. I can remember around May, that assessment was conducted. I can bear witness, and the special analysis result is based on that. And a lot of key activities were being planned to improve the uptake of this COVID-19. And I can assure you that we have seen the improvement in COVID-19 uptake in this state.” This underscores the targeted and impactful nature of capacity-building efforts initiated with the support of eHA to address specific gaps and enhance the effectiveness of COVID-19 initiatives.

Their rationale for considering this change most important is that the right capacity is critical for making the right professional judgments and providing the right leadership required for an effective vaccination program.

“If you don’t have enough capacity, there’s no way you can make a rational decision. And then they are at the field, working with what you told them. So, if you are wrong, you mislead everybody. If you are right and you know the thing, then everybody will be on the right track. So, building their capacity and closing the gap for the fragile and for those areas that have less capacity by the EHA has increased or has improved their decision-making. And that translates to all the things that we are doing in the field.” ...Cross-cutting respondent, Borno State

2) Increased government ownership and commitment to COVID-19 vaccination: Respondents attributed the increased ownership of COVID-19 vaccination by state actors to the support provided by the TA intervention. The respondents described that the state is now able to take a lead role in galvanizing the support of partners and can articulate its needs and priorities, engaging with partners to ensure better outcomes. A cross-cutting respondent in Akwa-Ibom articulated it this way: “So the project has provided us with this platform and given us some kind of added credibility because we were able to pull in other partners and say, look, the TA project is doing this and we have come out to you, this is our shopping list of support we require.” This emphasizes the role of the TA support project in not only providing direct support but also in empowering states to collaborate effectively with partners and take ownership of their COVID-19 vaccination efforts.

3) Strengthened supportive supervision system: As a result of eHealth Africa’s COVID-19 Technical Assistance (TA) project, states were able to implement a strengthened supportive supervision system, perceived by respondents to contribute to the quality of vaccine deployment initiatives. A respondent in the Federal Capital Territory (FCT) describes it this way: “Yes, the TA support project has been of great help. They are one of our supporting backbones of COVID-19 services. Number one, they have been so good; they have helped us so many times to do supportive supervision.” This highlights the positive impact of TA’s support in reinforcing the supervision system, ultimately contributing to the effectiveness and quality of COVID-19 vaccination initiatives.

4) Strengthened financial management system: One respondent from the Federal Capital Territory (FCT) attributed the improved financial management system of the FCT Primary Health Care (PHC) Board to the support of eHealth Africa. She mentioned, “The accounting software that was also introduced to us because it was also part of the... So, our accountants, their capacity building was also done.” This emphasizes the contribution of TA’s support not only in programmatic aspects but also in strengthening financial management systems within relevant government agencies, thereby enhancing overall efficiency and accountability.

5) Capacitating the states towards routinization of COVID-19 vaccination: Most of the states highlighted the challenges associated with transitioning from campaign style to routine vaccination through the health facilities. The campaigns heavily relied on ad-hoc staff. Routinization requires the availability of adequate human resources at the health facility and involves initial capacity building.

Improved resourcing of COVID-19 vaccination was achieved by galvanizing the support of multiple partners and through direct funding of activities and training. By strengthening the TWG, states were able to garner the financial support of all partners, in addition to the direct financial support provided by the COVID-19 TA project. The respondents also mentioned the increasing commitment of the government, reflected in the increased allocation from 6% to 17% and the recruitment of more health workers to support the routine at the health facility.

“The government now, they have been able to allocate more resources. From 6% to 17% of the budget for the year, and also, release emergency funds.” ...Cross-cutting Respondent, Borno State

This outcome involves the institutionalization of a review process and the promotion of statewide accountability for vaccination planning and performance. In Borno State, field-level supportive supervision has been incorporated into the activities of the Technical Working Group (TWG). The functional coordination structure has provided a platform that promotes a culture of regular progress and performance review using data. A respondent from Oyo State emphasized this institutionalized review, stating, “There is a lot of improvement. At the SERICC meeting, what we try to do is to look at data and challenges people are having in the field across the Local Government Areas (LGAs) and try to solve them in the meeting.” The ongoing review of progress and follow-through on action points have been linked to improved coordination of COVID-19 vaccination, fostering teamwork within the TWG.

The TA-CVDP operated on the premise that strengthening COVID-19 TWG coordination meetings, improving data utilization for vaccine program planning, and addressing institutional gaps in the vaccine supply chain processes would collectively lead to an enhancement in Nigeria’s COVID-19 vaccination coverage (Figure 1). All states have independently reported significant advancements in COVID-19 vaccination coverage. For instance, Ebonyi showcased considerable progress, emerging from the lower ranks of states, with a respondent noting, “And I know we have improved. We are not second to last again.” Similarly, Akwa-Ibom’s cross-cutting respondent expressed a positive shift from being among the bottom states to a more favorable mid-range position. As articulated by a cross-cutting respondent, “Obviously, I would say we have because, you know, we’re looking at where we are now. We were found among the last three states, struggling between the last state; it is always, you know, second to last or third to last. I think at the moment now, about 50%, that’s where we are—halfway up there.”

Figure 1. Scope of work as a research assistant.

5. Discussions

This study found that the COVID-19 Vaccine Deployment Technical Assistance (TA) project significantly improved the coordination of interventions and teams in multiple states. Participants from Borno and Katsina emphasized the comprehensive support from the TA project, which facilitated the integration of COVID-19 issues into other existing health structures, such as the State Emergency Routine Immunization Committee (SERIC) and the State Emergency Maternal and Child Health Intervention Centre (SEMCHIC). These findings align with existing literature that shows the importance of integrated health approaches for effective responses to public health emergencies (Jones et al., 2021). The enhanced coordination contributed to more efficient use of resources and better stakeholder engagement, which are crucial for successful vaccination campaigns (Brown et al., 2019).

The introduction of an action plan tracker was another initiative that brought about significant change, leading to better tracking of Technical Working Group (TWG) action plans. Respondents from all six states reported that this tool improved documentation and follow-through on action points. This finding is consistent with several other studies highlighting the benefits of digital tools in enhancing the efficiency and accountability of health interventions (Kowatsch & Fleisch, 2021; Krukowski et al., 2024; Laurie et al., 2021). The action plan tracker not only facilitated better meeting management but also ensured continuity and shared responsibility among stakeholders, which is critical for sustained public health efforts. A technical assistance project aimed at strengthening the health system during the COVID-19 pandemic also reported that tracking tools were instrumental in aligning stakeholders and optimizing resource use in Ghana (WHO, 2020b).

The routinization of evening debrief meetings during immunization campaigns was particularly noted in Akwa Ibom state. This practice allowed for real-time problem-solving and adjustments, significantly enhancing the effectiveness of campaigns. Literature supports the effectiveness of regular debriefs in improving operational efficiency and response times in health interventions (Kolbe et al., 2021; Allen et al., 2018). The technical support provided by the TA in organizing these meetings proved vital in addressing immediate challenges and optimizing resource allocation.

The reactivation or increased functionality of Technical Working Groups (TWGs) across five states highlights the pivotal role of regular coordination meetings facilitated by TA support. Regular meetings ensured proper planning, synergy among various actors, and timely reviews of progress, addressing emerging challenges effectively. Studies have shown that well-coordinated TWGs are essential for the success of health programs, providing a platform for strategic planning and collaborative problem-solving, which emphasizes that enhanced coordination among health partners and stakeholders significantly improves vaccination campaign outcomes (Sakala et al., 2023; Sharan et al., 2022).

The project’s impact on data utilization is noteworthy, particularly in achieving more accurate population estimates and enhancing data-driven decision-making. Research corroborates these findings, illustrating the importance of accurate data for effective vaccination strategies. A study demonstrated that data triangulation and integration of multiple data sources significantly improved the accuracy of population estimates and vaccination coverage in Nigeria (Rachlin et al., 2024). The project’s introduction of performance tracking and peer-review mechanisms has also been shown to drive improvements in vaccination coverage. A performance management review study in vaccination programs found that data-driven approaches and performance comparisons fostered competition and accountability, leading to increased vaccine uptake and coverage (Patel et al., 2023).

Institutional strengthening has been a key focus of the TA project, with respondents noting improvements in decision-making and capacity building. This aligns with broader research on institutional support in health systems. Capacity-building initiatives and institutional support were critical for enhancing health system performance and sustainability (Mghamba et al., 2023). The reported increases in vaccination coverage across states are consistent with findings from other studies. For instance, a comprehensive review of vaccination program interventions found that strengthening coordination, leveraging data, and enhancing institutional capacity were key factors in improving vaccination coverage during the COVID-19 pandemic (Hopkins et al., 2023).

Consistent with the project theory of change, respondents attributed the increased vaccination coverage in the supported states to the TA’s interconnected efforts at ensuring functional coordination structures for COVID-19 vaccination; promoting availability and use of good quality data for vaccination program management; as well as addressing institutional gaps in the state, which have played critical roles in improving COVID-19 vaccination coverage in the supported states.

The study’s findings corroborate the significant positive impact of the COVID-19 Vaccine Deployment Technical Assistance project across various states. Enhanced coordination, improved tracking of action plans, regular debrief meetings, and reactivation of TWGs have collectively contributed to more effective COVID-19 vaccination efforts. These findings underscore the importance of integrated support, digital tools, and regular coordination in public health interventions. Future projects should focus on maintaining these gains and addressing gender disparities to ensure inclusive and sustainable health outcomes.

6. Conclusion

This most significant change evaluation of the COVID-19 Vaccine Deployment Technical Assistance (TA) project, derived from insights shared by project stakeholders, robustly affirms its positive impact on various facets of the vaccination initiative. In summary, the TA-CVDP not only successfully achieved its three primary objectives which are enhancing the planning and coordination of COVID-19 vaccination efforts, improving data utilization for planning and vaccination actions, and strengthening institutional frameworks, but also substantiated, by its theory of change, that these objectives have plausibly contributed to the notable improvements observed in COVID-19 vaccination coverage, as perceived by stakeholders and supported by triangulated qualitative evidence.

While stakeholders attributed many of the observed improvements to the Technical Assistance (TA) intervention, it is important to acknowledge plausible alternative explanations. These include the concurrent involvement of other development partners (e.g., WHO, UNICEF), broader national and state-level policy shifts, increased government funding and ownership, and improvements in vaccine supply and logistics over the same period. To partially account for these factors, the analysis triangulated perspectives across multiple respondents, states, and domains, and examined consistency in reported mechanisms of change linked specifically to TA-supported activities (e.g., action plan trackers, data use capacity building, and TWG strengthening). However, given the complexity of the health system context and the qualitative nature of the MSC methodology, the findings should be interpreted as indicative of contribution rather than exclusive attribution.

Acknowledgments

The authors wish to acknowledge the support received from Peace Ndubuisi-Brown, Aisha Linatu Abdulrahman, Arit Udoakagha, ThankGod Nyeche, Akorede Akerele, Abubakar Ladan, Tijesu Ojumu, Onyeka Onwuama, and Dr. John Ibitoye.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References