Sample size determination typically relies on a power analysis based on a frequentist conditional approach. This latter can be seen as a particular case of the two-priors approach , which allows to build four distinct power functions to select the optimal sample size. We revise this approach when the focus is on testing a single binomial proportion. We consider exact methods and introduce a conservative criterion to account for the typical non-monotonic behavior of the power functions, when dealing with discrete data. The main purpose of this paper is to present a Shiny App provid ing a user-friendly, interactive tool to apply these criteria. The app also provides specific tools to elicit the analysis and the design prior distributions, which are the core of the two-priors approach.
Sample Size Determination (SSD) is an essential step in the design of a research study, especially in clinical trials. Let us denote by θ the parameter of interest, which measures the efficacy of a novel treatment, and assume that we are interested in testing H 0 : θ ∈ Θ 0 vs H 1 : θ ∈ Θ 1 , where Θ 0 and Θ 1 form a partition of the parameter space Θ .
A well-established strategy for SSD exploits the concept of power function: the study is sized to guarantee a large probability of rejecting the null hypothesis H0, when it is actually false. The decision about the rejection of H0 can be made under a frequentist framework or by performing a Bayesian analysis. In this latter case, a prior distribution, called the analysis prior, is introduced to incorporate in the procedure pre-trial knowledge the researcher wants to take into account, together with pre-experimental evidence if available. Moreover, the conjecture that the alternative hypothesis is true represents an essential element of the methodology. It can be realized by assuming that the true θ is equal to a fixed design value θ D , suitably selected under H1 and, in this case, the probability of rejecting H0 is evaluated by exploiting the sampling distribution of the test statistic conditional on θ D (conditional power). Alternatively, we can introduce uncertainty on the guessed design value by incorporating another prior distribution, called the design prior, which assigns negligible probability to values of θ under H0. In this latter case, the probability of rejecting H0 is computed by exploiting the prior predictive distribution of the test statistic under the assumption that θ is distributed according to the design prior (predictive power). By combining frequentist and Bayesian procedures of analysis, with both the conditional and the predictive approach, we can obtain four power functions that we can use for sample size determination (see [
In this paper, we consider the problem of SSD based on power analysis when the focus is on single-arm studies based on a single binomial proportion. This design is typically used in Phase II of clinical trials, where the parameter of interest is the probability of response to a novel therapy. Sambucini [
In practice, when the interest is focused on a single binomial proportion, many software tools have been developed to implement the standard procedures for SSD, based on power analysis conducted using the frequentist conditional approach. Instead, to our best knowledge, up until now, no software tool has been available to implement exact criteria based on the other three power functions. Thus, we developed an R Shiny App [
The rest of the paper is organized as follows. In Section 2, we revise the exact procedures, based on the two-priors approach, to select the optimal sample size for a single binomial proportion. Section 3 discusses some strategies to elicit the prior distributions. In Section 4, we present the Shiny App and illustrate its features through an example. Finally, Section 5 contains some concluding remarks.
In this Section, we revise the exact SSD procedures based on four possible power functions, assuming that interest is on one-sample testing problems with a binary response [
Specifically, let us suppose that we are interested in testing the proportion of responders to a novel therapy. We consider n patients, each of whom receives the same treatment dosage, and classified as responders or not to the therapy by using a binary variable Y. We assume that we are interested in testing H 0 : θ = θ 0 vs H 1 : θ > θ 0 , where θ denotes the parameter of interest, i.e. the true response rate, while θ 0 is a fixed target value that should represent the efficacy rate of the standard of care and is usually estimated through historical data.
Initially we assume that, at the analysis stage, the decision of rejecting the null hypothesis is made under a frequentist framework. The test statistic to use is the number of responders out of the n patients, Yn, whose sampling distribution of is f n ( y n | θ ) = bin ( y n ; n , θ ) , for y n = 0, ⋯ , n , where bin ( ⋅ ; n , p ) denotes the probability mass function of a binomial with parameters n and p. Therefore, the frequentist rejection region at level α is R H 0 = { y n ∈ { 0,1, ⋯ , n } : y n ≥ r } , where the critical value r is given by
r = min { k ∈ { 0,1, ⋯ , n } : ∑ i = k n bin ( i ; n , θ 0 ) ≤ α } . (1)
Note that, since the binomial distribution is discrete, the actual Type I error rate does not hit α exactly, but it is always less than or equal to it. In order to exploit the power function for SSD purposes, at the design stage, we need to consider a scenario under which the alternative hypothesis is true. A first possibility is to specify a design value θ D for θ that belongs to H1. In this case, we obtain the frequentist conditional power η F C ( n ) = ℙ f n ( ⋅ | θ D ) ( Y n ∈ R H 0 ) = ∑ y n = r n bin ( y n ; n , θ D ) , where ℙ f n ( ⋅ | θ D ) denotes the probability measure associated with the sampling distribution of Yn for θ = θ D . Note that η F C ( n ) represents the probability of correctly rejecting H0 using a frequentist procedure, when θ is equal to θ D . However, we can add flexibility to the procedure by avoiding the use of a fixed design value. In fact, it is possible to introduce uncertainty on the suitable design value to specify by eliciting a design prior distribution, π D ( θ ) . This latter is an instrumental tool that allows to model design expectations on θ , under the assumption that the treatment is effective. Consequently, it should be chosen as an informative distribution, as we will discuss further in Section 3. In our specific case, given a beta design prior, π D ( θ ) = beta ( θ | α D , β D ) , the prior predictive distribution of Yn is
m n D ( y n ) = beta-bin ( y n ; α D , β D , n ) , for y n = 0, ⋯ , n , (2)
where beta-bin ( ⋅ ; p , q , n ) is the probability mass function of a beta-binomial with parameters p, q and n. Therefore, the frequentist predictive power is given by η F P ( n ) = ℙ m n D ( ⋅ ) ( Y n ∈ R H 0 ) = ∑ y n = r n beta-bin ( y n ; α D , β D , n ) where ℙ m n D ( ⋅ ) denotes the probability measure associated with the prior predictive distribution of Yn in (2) and r is the critical value defined in (1). Note that η F P ( n ) provides the probability of correctly rejecting H0 using a frequentist procedure, when θ is guessed to belong to the alternative hypothesis, where it is distributed according to π D ( θ ) .
Alternatively, it is possible to perform the analysis under a Bayesian framework. This choice allows us to take into account pre-experimental information available on the treatment, for instance, based on historical data or on the subjective opinions of experts. The information is incorporated through the elicitation of another prior distribution on the parameter, the analysis prior distribution, π A ( θ ) . By exploiting conjugate analysis results, we consider a beta density, π A ( θ ) = beta ( θ ; α A , β A ) , so that the corresponding posterior distribution is
π n A ( θ | y n ) = beta ( θ ; α A + y n , β A + n − y n ) .
Under this setup, to build a Bayesian equivalent of a power function, we need to determine the set of values of Y n that, if observed, would lead to rejecting the null hypothesis. In line with Spiegelhalter et al. [
ℙ π n A ( ⋅ | Y n ) ( θ > θ 0 ) > 1 − ε (3)
where ℙ π n A ( ⋅ | Y n ) denotes the probability measures associated with the posterior distribution of θ and ε is a probability threshold typically selected as a small value. The condition in (3) is a random object at the design phase because it depends on the future result Yn. Nevertheless, for a fixed value of n, the probability that it is fulfilled increases with Yn. Therefore, the Bayesian rule consists in rejecting H0 if Y n ≥ r ˜ , where
r ˜ = min { k ∈ { 0 , 1 , ⋯ , n } : ℙ π n A ( ⋅ | k ) ( θ > θ 0 ) > 1 − ε } . (4)
Then, we need to compute the probability that the Bayesian significance condition is fulfilled under the optimistic assumption that the treatment is effective. Once again, we may realize this assumption by using either a conditional or a predictive approach. In the first case, we fix a suitable design value θ D under H1 and define the Bayesian conditional power as
η B C ( n ) = ℙ f n ( ⋅ | θ D ) [ ℙ π n A ( ⋅ | Y n ) ( θ > θ 0 ) > 1 − ε ] = ∑ y n = r ˜ n bin ( y n ; n , θ D ) .
In the second case, we elicit a design prior distribution for θ , as described above, and obtain the Bayesian predictive power, that is
η B P ( n ) = ℙ m n D ( ⋅ ) [ ℙ π n A ( ⋅ | Y n ) ( θ > θ 0 ) > 1 − ε ] = ∑ y n = r ˜ n beta-bin ( y n ; α D , β D , n ) .
Clearly, both the power functions η B C ( n ) and η B P ( n ) provide the probability of correctly rejecting H0 using a Bayesian procedure, under the assumption that the alternative hypothesis is true. Moreover, it is worth pointing out that the Bayesian predictive power is the one that allows to model both prior knowledge and uncertainty on the design value: it includes the other power functions as special cases. In fact, if we consider a point-mass design distribution on θ D , then no design uncertainty is involved, and the predictive approach coincides with the conditional one. On the other hand, If no pre-experimental information is available, a non-informative analysis prior can be elicited and the Bayesian powers coincide with the frequentist one.
Whatever the power function chosen, the standard SSD criterion selects the optimal n as the minimum value such that the power exceeds a threshold of interest γ . Hence, the optimal sample sizes are obtained as
n J I = min { n ∈ ℕ : η J I ( n ) ≥ γ } , I = C , P , J = F , B . (5)
where the superscript refers to the approach used at the design stage, while the subscript refers to the approach used at the analysis stage. However, given the saw-tooth shape of the power curves as a function of n, a slightly different and more conservative SSD criterion can be adopted. The idea is to select the smallest sample size such that the condition on the power is fulfilled also for all the sample size values greater than it, that is
n J I = min { n * ∈ ℕ : η J I ( n ) ≥ γ , ∀ n ≥ n * } , I = C , P , J = F , B . (6)
This latter criterion prevents the condition of interest from being satisfied for the selected sample size, but no longer satisfied for some larger values of n.
This section discusses some strategies to elicit the design and analysis prior distributions, accounting for their different aims. We start focusing on the design prior distribution π D ( θ ) . The idea is to express the hyperparameters in terms of the prior mode θ D and the prior sample size n D by using [
α D = n D θ D + 1 and β D = n D ( 1 − θ D ) + 1. (7)
We can center π D ( θ ) on the design value we would consider in the conditional approach and regulate the concentration through the choice of the prior sample size. It is crucial to emphasize that π D ( θ ) should be an informative distribution. First, it serves to realize the assumption that θ belongs to the alternative hypothesis. Furthermore, as n approaches infinity, η B P ( n ) and η F P ( n ) tend to the probability assigned to the alternative hypothesis by π D ( θ ) , denoted by ℙ π D ( ⋅ ) ( θ > θ 0 ) [
1) π D ( θ ) assigns a probability close to one to the alternative hypothesis;
2) π D ( θ ) assigns a probability close to one to a symmetric interval ( θ D − δ , θ D + δ ) , where δ is a non-negative real number such that θ D − δ ≥ θ 0 .
Both these procedures are implemented in the Shiny App described in the next Section. Finally, as n D tends to infinity, π D ( θ ) tends to assign all the probability mass to the prior mode θ D , resulting in no variability introduced around it. Consequently, the predictive and conditional approaches align.
The elicitation of analysis prior distribution can be based on historical data or on subjective opinions of experts. However, one of the most common ways of proceeding is to choose a non-informative density or a density based on very weak information, to avoid the possibility of underweighting the results of the current experiment in determining the analysis outcome. Thus, for instance, we may rely on the Uniform distribution on the interval ( 0,1 ) by specifying α A = β A = 1 or on the non-informative Jeffreys prior by setting α A = β A = 0.5 . As an alternative and similarly to the choice of π D ( θ ) , we can express the hyperparameters of the analysis prior distribution in terms of prior mode, θ A and prior sample size n A , where n A is typically fixed equal to one, or equal to a very low value, in order to obtain a weakly informative prior distribution. This way of proceeding allows to express skepticism, neutrality or optimism about large treatment effects through the choice of the prior mode θ A . Finally, let us notice that if we introduce no prior information, i.e. if n A is set equal to 0, the Bayesian and the frequentist setup coincide.
This Section presents a Shiny App that implements the sample size criteria described in Section 2. It is available at the following link: https://susanna-gentile.shinyapps.io/SSD_singlearm.
The Shiny package in R enables the creation of interactive web apps directly from R ( [
To allow computing the optimal sample size according to the four power functions;
To implement both the standard and the conservative criterion to select the optimal sample size;
To display, if requested, the power function behaviour as a function of n;
To enable storing the design parameters and results into a table and to download it as a CSV file;
To help to select the analysis and the design prior distributions and to visualize them.
Thus, the users can select either the conservative criterion in (6), accounting for the saw-tooth behaviour, or the standard criterion. We suggest using the first criterion. However, we let this choice be at the user’s discretion as there is no unanimous agreement on the appropriateness of this methodology [
We start by describing the User Interface (UI). The UI changes accordingly to the methodologies chosen to conduct the design and analysis phases, as depicted in Figures 1-4. In the upper part of the UI, users can input the design parameters, split into three groups.
General setting: The inputs include the historical control θ 0 , the power level γ , and the maximum sample size. Users can choose whether to use the conservative criterion (the default), the standard one, or both.
Analysis stage: The inputs depend on the planned final analysis. For a frequentist analysis, the app requires the Type-I error probability α (
Design stage: The inputs depend on the approach used to realize the optimistic assumption that the experimental treatment is effective. The app requires the design value θ D for the conditional approach (
Once all the required inputs have been provided, the “Results” Panel prints on the left a summary of the design parameters, the optimal sample size, and the critical value. On the right, if requested, the power as a function of n is displayed. Users can save the results in a table by clicking “Save results”. The table can then be downloaded as a CSV file (
Note that if the user changes the target value parameter θ 0 , he will have to insert the other design parameters again. This mechanism prevents the app from crashing if the old input parameters are inconsistent with the new target value and the corresponding hypothesis system. Once the user provides θ 0 , the app will check if all the inputs satisfy the boundaries before computing the results and inform the user otherwise.
The maximum sample size input allows the user to specify the maximum sample size available. If the optimal sample size exceeds the maximum, the app prints a message and the power corresponding to the maximum sample size in the “Result” panel. By default, the app still computes the optimal sample size according to the selected criterion, as shown in the plot on the right. However, suppose the optimal sample size is greater than 1000. In that case, the app stops, and the user can decide whether to increase the maximum sample size by modifying the input parameter, as suggested by a printed message. This boundary also prevents the app from crashing if the optimal sample size at the desired level does not exist, which may happen if the prior distributions are not well specified.
The “Design prior” panel appears if the user opts for the predictive approach at the design stage (
By assigning a fixed probability to the alternative hypothesis:
the resulting π D ( θ ) assigns the selected probability to the alternative hypothesis.
By assigning a fixed probability to an interval:
the resulting π D ( θ ) assigns the selected probability to a symmetric interval [ θ 0 − δ , θ 0 + δ ] .
Manually: the user can select the prior sample size.
As emphasized in the previous section, the design prior should be highly informative and assign a negligible probability to the null hypothesis. The first two methods implement the strategies described in the previous Section and ensure this condition by selecting n D numerically. However, users can also choose n D at their discretion. In this case, we encourage users to verify if the probability assigned to the alternative hypothesis is greater than the desired power level γ . To help the user in the choice, this probability corresponding to the inserted n D is printed under the hyperparameters. Regardless of the selected method, the design distribution is displayed on the right. Finally, the “Update prior parameters” button allows for updating the corresponding inputs in the user interface with the hyperparameters of the chosen distribution.
The “Analysis prior” panel appears when the user decides to conduct the analysis stage under a Bayesian framework (see
1) The hyperparameters α A and β A are both greater than 1, so that π A ( θ ) admits a mode in θ A ;
2) The prior sample size n A is less than 100.
As outputs, the panel returns the hyperparameters and a graphical representation of the analysis prior. If the user opts for selecting n A manually, the probability assigned to the alternative hypothesis is also shown for a check. If the user opts for the automatic selection, the app prints the prior sample size n A instead. Finally, α A and β A can be stored in the corresponding UI values by clicking on “Update prior parameters”.
We now illustrate an example of the app utilization. When inserting the inputs, we suggest starting from the target value θ 0 so that the app can automatically check if the other parameters are well selected.
Let us start by considering the frequentist conditional power; the corresponding UI is shown in
Then, we switch to a predictive approach by selecting “Predictive” in the Design Stage window. This choice leads to the User Interface in
Let us suppose now that there is an optimistic prior opinion toward the treatment efficacy, and the most plausible value for the parameter, according to experts, is θ = 0.3 . We switch to a Bayesian analysis framework to incorporate this information.
The methodologies based on the two-priors approach allow to exploit four different power functions to determine the optimal sample size. We revise these procedures when the focus is on single-arm studies based on a single binomial proportion. Although there are several R packages and software tools to implement the classical procedures for SSD, based on the frequentist conditional approach, easy-to-use computational tools to implement criteria based on the other three power functions are not yet available. To fill this gap, we developed an interactive and used-friendly Shiny application, whose main functionalities are presented in this paper.
In addition to allowing the calculation of the optimal sample size, the Shiny app allows us to check the behaviour of the four power functions as the sample size varies and let the users choose between the standard and the conservative criterion for SSD, which takes into account the saw-tooth behaviour of the power functions. Moreover, since the distinction between analysis and design priors is an essential element of the implemented procedures, the app provides two separate panels to help the selection of both the prior distributions: different strategies to elicit these priors can be used and the app allows us to visualize the corresponding plot.
Finally, let us notice that in this paper we refer to single-arm designs in phase II of clinical trials. These designs are frequently used to determine whether a new treatment is likely to meet a basic level of efficacy, before comparing it with the standard therapy in larger and randomized phase III trials, and the efficacy is commonly measured as a response rate. However, the SSD procedures implemented in the Shiny App we present can be used to size experiments conducted in fields other than the clinical one, as long as based on a single binomial proportion.
The authors declare no conflicts of interest regarding the publication of this paper.
Gentile, S. and Sambucini, V. (2024) Frequentist and Bayesian Sample Size Determination for Single-Arm Clinical Trials Based on a Binary Response Variable: A Shiny App to Implement Exact Methods. Open Journal of Statistics, 14, 90-105. https://doi.org/10.4236/ojs.2024.141004