Using Real-Time Quantitative Data Saturation Analysis to Enhance the Rigor of Qualitative Research: A Decision-Making Perspective ()
1. Introduction
Morse (2015) described the state of the art in determining the necessary sample size for a qualitative study to produce meaningful results as follows: “Saturation is the most frequently touted guarantee of qualitative rigor offered by authors to reviewers and readers, yet it is the one we know least about” (p. 587). The purpose of this paper is to address this serious problem affecting the credibility and applicability of qualitative research study findings, and thus their usefulness to decision makers. In this paper, I examine the nature and current state of data saturation analysis and explain the principles and application of a method of real-time quantitative data saturation analysis designed to address the problem.
2. Sample Size and Data Saturation
The number of interviews necessary to ensure a high level of study rigor is the fundamental question qualitative researchers who choose this method of data collection must answer (Baker & Edwards, 2012; Braun & Clarke, 2006; Fusch & Ness, 2015; Robinson, 2022). The answers range from a simple “it depends” to highly divergent methods prescribed for determining sample size that are more art than science. One accepted requirement of the data collection process is the need to reach saturation before declaring sufficient data exists to confidently answer the research question(s) of the study. Unfortunately, “saturation is the most frequently touted guarantee of qualitative rigor offered by authors to reviewers and readers, yet it is the one we know least about” (Morse, 2015: p. 587).
According to Saunders et al. (2018: p. 1896), qualitative researchers employ four primary approaches to saturation—inductive thematic, a priori semantic, theoretical, and data saturation. Data saturation “relates to the degree to which new data repeat what was expressed in previous data” (p. 1897). It is the point in the process at which no new themes emerge from the data, indicating data collection is complete.
3. Data Saturation Analysis
Hennink and Kaiser (2022) examined two types of data saturation guidelines—empirical and theoretical—based on 17 studies by authors who used empirical data to assess saturation and 6 who used statistical modeling. They found that “studies using empirical data reached saturation within a narrow range of interviews (9 - 17)… particularly those with relatively homogenous study populations and narrowly defined objectives” (p. 1). This supports Guest et al.’s (2006) finding that “theme saturation—defined as identifying around 90% of the themes in a set of qualitative interviews—could be reached within 6 - 12 interviews” (Wutich et al., 2024: p. 1), and Hennink et al. (2017), who stated that “saturation was reached at nine interviews, whereby the range of thematic issues was identified” (p. 591). In studies involving the use of statistical models, the sample size estimates for saturation were similar to those in studies using empirical data (Hennink & Kaiser, 2022). The problem with using either a priori empirical guidelines or statistical estimates of the point of data saturation is that neither is grounded in the actual study data. In the rest of the paper, I address this serious weakness in qualitative research practice by providing, justifying, and illustrating the application of an alternative, real-time quantitative method for determining data saturation appropriate for many types of qualitative research.
4. Real-Time Quantitative Data Saturation Analysis: Principles and Practice
This section includes a definition of “real-time” appropriate to the qualitative research data collection and analysis process, a justification of why real-time quantitative data saturation analysis is necessary, and a discussion of the principles and practice of real-time quantitative data saturation analysis.
4.1. What Does Real-Time Mean in Data Saturation Analysis?
Merriam-Webster (n.d.) defines real time as “the actual time during which something takes place.” Real-time data saturation analysis refers to the time during which the qualitative data collection and analysis process takes place.
4.2. Why Use Real-Time Quantitative Data Saturation Analysis?
The procedures for qualitative data analysis are more interpretive in nature than those for analyzing quantitative data, suggesting they are more art than science. In essence, they lack the degree of transparency typical of quantitative approaches, which reduces the credibility of qualitative research findings, and they are more prone to researcher bias because of their interpretive nature, which reduces the quality of the research findings.
Transparency
“Transparency is a defining characteristic of all scientific endeavours” (Sampson et al., 2019: p. 2). On the subject of transparency, Little (1970: p. 1841) argued, “People tend to reject what they do not understand.” Unfortunately, qualitative research is often more opaque than transparent due to “researchers having the option to choose from an increasing array of theoretically and technically sophisticated methods” (Sandelowski, 2000: p. 335).
Bias
Many forms of bias affect the collection and analysis of qualitative data. Participant bias and researcher bias are the two most important to acknowledge and manage to ensure the quality and applicability of the findings of qualitative studies involving the collection and analysis of interview data.
Two primary sources of bias in qualitative studies stem from the perspectives of the participants and the researcher. On a scale from low to high potential bias, an individual participant’s perspective varies across the spectrum depending on the nature of the participant’s lived experiences; whereas, the bias of an overall group of study participants is lower due to the “aggregation” of participant data involved in qualitative data analysis. Potential researcher bias ranges from low to high due to variations in their prior theoretical knowledge and/or lived experience of the phenomenon being studied. A third cause of bias in qualitative research related to researcher’s personal biases results from their choice of research method. Basically, descriptive methods, which focus on the participant’s stated views are less susceptible to researcher bias than interpretive methods, which involve to a much higher degree the researcher’s interpretation of the meaning of the participant’s stated views (Sandelowski, 2000).
As described in the next section, the use of real-time quantitative data saturation analysis can significantly increase transparency and reduce researcher bias in qualitative research studies involving the collection and analysis of participant interview data, a standard method in qualitative research.
4.3. What Are the Principal Elements of Real-Time Quantitative Data Saturation Analysis?
The four principal elements of real-time quantitative data saturation analysis are: 1) purposeful selection of study participants, 2) precise alignment between the interview questions and the research question(s), 3) use of descriptive, rather than interpretive, data analysis, and 4) use of thematic analysis.
Purposeful Participant Selection for a Homogeneous Sample
Campbell et al. (2020) argued that the reason for purposeful sampling is “better matching of the sample to the aims and objectives of the research, thus improving the rigour of the study and trustworthiness of the data and results” (p. 653). The goal is to identify a set of participants (i.e., subject matter experts) with similar characteristics to ensure the collection of meaningful, comparable data.
Alignment of Interview Questions with Research Questions
To ensure study participants’ provide the data needed to answer a given research question, which is the goal of data collection, it is essential to align the related interview questions with that research question. For example, to learn about the participants’ lived experiences of a phenomenon, like resistance to change, the researcher should ask them a direct interview question about why they resist change, rather than a number of indirect questions designed to provide the information necessary for the researcher to interpret (i.e., infer) why the participants resist change. The reason for this is to keep to a minimum the degree of inadvertent researcher bias due to their preconceptions based on education, personal experience, or prior academic research on a topic.
Descriptive vs Interpretive Data Analysis
According to Sandelowski (2000: p. 235), all inquiry entails description, and all description entails interpretation. Knowing any phenomenon (or event or experience) requires, at the very least, knowing the “facts” about that phenomenon… Researchers seeking to describe an experience or event select what they will describe and, in the process of featuring certain aspects of it, begin to transform that experience or event.
The goal is to keep the level of researcher interpretation, and thus potential bias, of the data as low as possible. To that end, a descriptive approach, coupled with a strong focus on the participants as the subject matter experts, rather than the researcher, is essential to the quality of the real-time quantitative data saturation analysis method.
Thematic Analysis
According to Kiger and Varpio (2020: p. 847), “Thematic analysis is a method for analyzing qualitative data that entails searching across a data set to identify, analyze, and report repeated patterns.” Kiger and Varpio also noted that “a distinguishing feature of thematic analysis is its flexibility to be used within a wide range of theoretical and epistemological frameworks, and to be applied to a wide range of study questions, designs, and sample sizes.” Hence, the real-time quantitative data collection and analysis method involves applying “the principles of thematic analysis of how to code data, to search for and refine themes, and to report findings” (Kiger & Varpio, 2020: p. 847).
In contrast to the theoretical perspective presented in the section to justify the elements of real-time quantitative data collection and analysis, the focus of the next section is practical, containing examples of how to apply the method to identify potential themes, determine the point of data saturation, and decide upon the study themes.
5. How to Apply Real-Time Quantitative Data Saturation Analysis
This section contains a description of the real-time quantitative data saturation analysis method, including the iterative data collection and analysis process and the criteria for determining the point of data saturation beyond which further data collection and analysis are unnecessary, as well as guidelines for researchers using the method.
Data Collection and Analysis: Process Overview
The real-time quantitative data collection and analysis method consists of: 1) the standard qualitative research process of collecting and analyzing the data from each individual participant’s interview separately to identify potential themes, coupled with 2) quantitative analysis of the cumulative data from the participants interviewed to any given point in the process with the objective of determining if data saturation has occurred.
Data Saturation Analysis: Stopping Point
In real-time quantitative data saturation analysis, the researcher stops collecting and analyzing interview data when additional interviews do not yield any new codes/potential themes. The choice of how many interviews to conduct before ending data collection depends on the probability that yet another interview will yield additional codes/potential themes.
While it is not possible to compute this probability in advance, a researcher can select a stopping point for the interviews—the point of data saturation—in advance based on the on the following formula:
where, P(N) is the probability that no new codes/potential themes will emerge in N consecutive interviews, P(i) is the probability that no new codes/potential themes will emerge in a given interview from i equals 1 to N, and N is the number of consecutive interviews in which no new codes/potential themes have emerged.
For example, the value of P(N) for P(i) = 0.5 and N = 3 is:
Whereas, the value of P(N) for P(i) = 0.1 and N = 3 is:
Thus, for a reasonable probability that no new codes/potential themes will emerge in a given interview of 1 in 10, the probability of no new themes emerging over three consecutive interviews is 1 in 1000. Hence, a conservative stopping point for real-time quantitative data saturation analysis is three consecutive interviews in which no new codes/potential themes have emerged.
Real-Time Quantitative Data Saturation Analysis—Hypothetical Data
To illustrate the process of convergence on the study themes, this section contains a real-time quantitative data saturation analysis of hypothetical data collected from interviews with 6 participants. There are 3 study themes (Theme 1, 2, and 3) with probabilities of emerging from the data collection and analysis process of 0.30, 0.60, and 0.10. The three parts of the data collection and analysis process are:
Part 1—Data Saturation Analysis based on Potential Themes by Interview. As shown in the table below, Participant 1 (P1) mentioned Themes 1 and 2. Participant 2 also mentioned Themes 1 and 2, and Participant 3 mentioned Themes 1, 2, and 3. Participants P4, P5, and P6 each mentioned one theme (Theme 2), but no new themes. Hence, data saturation occurred after the first 3 interviews.
P1 |
P2 |
P3 |
P4, P5, P6 |
Theme 1 |
Theme 1 |
Theme 1 |
Each one Identified Theme 2 |
Theme 2 |
Theme 2 |
Theme 2 |
|
|
Theme 3 |
Note. The final themes are determined from these potential themes in Part 3 of the data collection and analysis process.
Part 2—Potential Themes by Interview in Rank Order. As shown in the table below, Theme 2, was the most frequently mentioned by the participants, followed by Theme 1 and Theme 3.
Theme |
P1 |
P2 |
P3 |
P4, P5, P6 |
# |
Prob. |
2 |
1 |
1 |
1 |
3 |
6 |
0.60 |
1 |
1 |
1 |
1 |
|
3 |
0.30 |
3 |
|
|
1 |
|
1 |
0.10 |
The table below, a distribution of the three potential themes, indicates convergence on Theme 2 as the most important theme.
|
Theme 2 |
|
|
Theme 2 |
|
|
Theme 2 |
|
Theme 1 |
Theme 2 |
|
Theme 1 |
Theme 2 |
|
Theme 1 |
Theme 2 |
Theme 3 |
Part 3—Analysis of Potential Themes by Interview in Rank Order to Determine the Study Themes. The table below of the potential themes rank ordered by the number of participants who mentioned each provides the data for determining the final study themes.
Theme |
P1 |
P2 |
P3 |
P4 |
P5 |
P6 |
# |
2 |
1 |
1 |
1 |
1 |
1 |
1 |
6 |
1 |
1 |
1 |
1 |
|
|
|
3 |
3 |
|
|
1 |
|
|
|
1 |
Logically, the study themes are those potential themes most often mentioned by the study participants; in this case, (potential) Themes 2, 1, and 3 in that order. However, because only one participant mentioned Theme 3, I would argue that it is a one-off and that (potential) Themes 1 and 2, representing 60% and 30% of the potential themes mentioned by the participants, respectively, are the themes that emerged from the real-time quantitative data saturation analysis of the study data.
Real-Time Quantitative Data Saturation Analysis—Real Data
Previously we examined a hypothetical case, in which the distribution of study themes was assumed to be known, to illustrate the process. This section contains a real-time quantitative data saturation analysis of actual data adapted from a recently published qualitative study to illustrate the process of determining the study themes from the list of potential themes identified in the data collection and analysis stage of the participant interview process.
The principal research question for the descriptive phenomenological study in question (Jimerson & Levasseur, 2025) was: What factors affect teachers’ motivation to remain teachers? The table below, which contains the themes that arose from the data collection and analysis arrayed in rank order by the number of participants who mentioned each. Data saturation occurred by the 7th of 10 interviews, as none of the final three participants mentioned any new potential themes. Potential themes mentioned by only one participant were treated as one-offs and not considered themes. Thus, the final eight study themes (i.e., factors affecting teachers’ motivation to remain teachers) in priority order were: 1) leadership support, 2) career purpose, 3) money/benefits, 4) strong team collaboration, 5) students’ behavior, 6) leadership micromanagement, 7) work/societal pressures, 8) and parents’ behavior.
Theme |
P1 |
P2 |
P3 |
P4 |
P5 |
P6 |
P7 |
P8 |
P9 |
P10 |
Total |
% |
Rank |
A |
1 |
|
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
9 |
90 |
1 |
B |
|
1 |
|
|
1 |
|
1 |
1 |
1 |
1 |
6 |
60 |
2 |
C |
1 |
1 |
|
|
|
|
1 |
1 |
|
1 |
5 |
50 |
3 |
D |
1 |
|
1 |
|
|
1 |
|
|
|
1 |
4 |
40 |
4 |
E |
|
1 |
|
1 |
|
|
1 |
|
|
1 |
4 |
40 |
4 |
F |
1 |
|
|
|
|
|
1 |
|
1 |
|
3 |
30 |
6 |
G |
1 |
|
|
|
1 |
|
|
|
|
|
2 |
20 |
7 |
H |
|
|
|
1 |
|
|
|
|
|
1 |
2 |
20 |
8 |
Note. Data saturation was reached as no new themes emerged from the interviews with participants 8, 9, and 10. Adapted from Jimerson and Levasseur (2025).
A Guideline for Researchers
From a decision-making perspective, the degree to which real-time quantitative data saturation analysis, when applicable, enhances the rigor of the qualitative research process and the resulting quality of the study findings depends primarily on the researcher’s focus. To increase the transparency of the process and reduce researcher bias, researchers should focus on what the participants said about a phenomenon based on their lived experiences rather than what they, as researchers, think the participants meant by what they said. In practice, given that qualitative interview data analysis requires some level of interpretation at the coding/potential theme identification stage, that means keeping additional interpretation to a minimum.
6. Conclusion
This paper is about what, why, when, and how of real-time quantitative data saturation analysis. This method applies to any qualitative research study based on the collection and analysis of data provided by participants in response to descriptive interview questions, which is often the case, for example, in phenomenological, grounded theory, and case studies. Like all research methods, real-time quantitative data saturation analysis has its limitations, but where it applies the study findings tie directly to the participants’ lived experiences, not the researchers interpretation of those lived experiences, thereby increasing transparency and reducing the likelihood of researcher bias. The end result is better research results on which to base better decisions.