For reverse engineering, we use both dynamic and static analysis techniques to gain detailed insights into the software:

Dynamic Analysis: The upper-left pipeline part in Figure 1 shows that we instrument the Fortran code with monitoring probes. We execute the ESMs utilizing configurations derived from climate science publications provided by our domain experts. The dynamic analysis observes the software at runtime and collects monitoring information about procedure calls. This approach detects which parts of software are actually used and how they behave. To instrument the ESM, we employ the monitoring framework Kieker [13] [14] in combination with the ability of the GNU Compiler Collection and Intel Fortran compilers to instrument functions utilizing the -finstrument-functions option.¹ Our dynamic architecture recovery tool dar², then recovers the architecture from the monitoring data.

Static Analysis: The lower-left pipeline part in Figure 1 shows that to perform the static analysis on Fortran code, we first pre-process the source code. To discover calls, dataflows, and accesses to common blocks from Fortran source code, we developed our fxca³ tool to produce lists of procedure calls, procedure name-to-file mappings, access to common blocks, and dataflows between procedures for the static architecture recovery tool sar⁴, which generates the statically recovered architecture model. fxca is an extension of fxtran⁵, which provides a concrete syntax tree of Fortran code.

While both dynamic and static analyses usually recover similar architectures, each one can identify certain aspects better than the other. The static analysis can identify common blocks in Fortran, while the dynamic analysis can collect information on procedures used from libraries, which the static analysis cannot provide with the same detail.

To take advantage of both analysis techniques, we then (optionally) merge the results of the static and dynamic analyses into a combined architecture model and discover module interfaces (right part of Figure 1 ).

In addition to control flow (i.e., the study of which operations are called from which points in the code), we also study dataflow, i.e., the question of which parts of the code communicate by accessing the same data. In Fortran, the default way of handling parameters in procedures is call-by-reference. This is used to pass data into subroutines and get results back. This has a significant impact on the dataflow analysis and the complexity of a system. In case a subroutine modifies call-by-reference parameters, a developer must keep side effects in mind when changing the program, while call-by-value operations have no side effects. A Fortran function is similar to a mathematical function, which takes parameter values as inputs and returns a single output value. A Fortran subroutine is a block of code that performs some operation on the input variables, and as a result of calling the subroutine, the input variables are modified (call-by-reference).⁶ When a distinction between functions and subroutines is not required for our analysis, we will use the term operation.

Since the question of whether data flows from one procedure to another via any means of communication (e.g., writing/reading the same file or memory location using array indexing or pointer arithmetic) is, in general, undecidable, we focus on dataflow using global variables which in Fortran are expressed as common blocks [15] and procedure calls. Procedure calls can have call-by-value and call-be reference parameters. Call by value is interpreted as a dataflow from the caller to the callee. For call-by reference, we check whether the reference is used for read or write operations within the procedure. In case only read operations are used, the dataflow is from the caller to the callee. Write operations result in a dataflow from the callee to the caller. In case both operations are used, the architecture model contains a bidirectional dataflow edge, and the analysis graphs provide two edges for each direction. In case there are multiple dataflows between two procedures, they are merged following the same principle as with calls for two reasons:

1) Developers have to consider the dependency between both procedures regardless of the number of parameters.

2) Parameters can have basic or composed data types.

Compound data types can be replaced by a set of parameters with basic types. Common blocks are seen as one single “piece of data,” and we do not distinguish between access to different variables in the same block. The reason for this decision is that we assume that the common block is usually seen as a single entity by the developers of the ESM, and therefore, two procedures that access the same common block are conceptually coupled. Our analysis scripts could also be used to analyze a more fine-grained view of the distinction between variables in the same block.

Fortran 77 does not provide syntactical structures that describe interfaces to modules, and even code conforming to Fortran 90 and above does not use this language feature in the two ESMs we analyzed. Thus, the recovered architecture, as initially obtained with Kieker, does not contain interfaces. To support the understanding of the architecture and the code, our second step infers interfaces based on cross-module calls.

Interfaces help developers understand which (public) operations are used by other modules and which (private) operations represent module-internal functionality. Well-designed interfaces group operations together that address a specific concern, e.g., writing and modifying files.

Therefore, we discover interfaces based on calls crossing module boundaries, i.e., calls that originate in one module and call operations in another module. Solely based on these inter-module calls, it is possible to generate candidates for interfaces in four different ways:

1) One large interface for each module pair, resulting in multiple interfaces for one module that may contain the same or similar sets of operations.

2) One interface for all exposed, i.e., externally called operations, per module, resulting in multiple other modules requiring the same interface. These interfaces can become very large and mingle many concerns together.

3) One interface per exposed operation, resulting in many interfaces, making it hard to group associated operations together.

4) Grouping operations together that have the same set of callers. This may still lead to numerous interfaces, but if developers follow a pattern when using operations, this approach will reduce the number of interfaces in contrast to Option 3.

For this paper, we applied Option 4 and implemented four steps to compute interfaces for an architecture description.

For the formal description of our interface discovery process, we use $O$ as the set of all operations in the ESM. The modules of the software are denoted $M_1$ to $M_n$. These modules constitute a partition of $O$, i.e., $O=M_1\cup \dots \cup M_n$ and $M_i\cap M_j=\varnothing$ for $i\ne j$. The function $module$ returns the corresponding module $o$ for any given operation and is defined as:

$module\left(o\right)=M_i\text{with}o\in M_i$

All calls between operations in the architecture model are represented by $E_{all}\subseteq O\times O$.

Step One: The first step creates a subset $E_{sub}$ of all distinct calls $E_{all}$ in the execution model where the caller $o_r$ and callee $o_p$ of a call are not in the same module:

$E_{sub}=\left\{\left(o_r,o_p\right)|\left(o_r,o_p\right)\in E_{all}\wedge module\left(o_r\right)\ne module\left(o_p\right)\right\}$

Step Two: In this step, we identify all provided operations of one module $M_j$ in relation to a requiring module $M_i$. This is realized by the following function taking the respective indices $i$ and $j$ of the modules. This function $provided\text{\_}ops$ returns the subset of $M_j$ representing the provided operations. This resembles Option 1:

$provided\text{\_}ops\left(i,j\right)=\left\{o_j\in M_j|\exists o_i\in M_i\exists \left(o_i,o_j\right)\in E_{sub}\right\}$

Step Three: As the previous step creates one provided interface for each requiring module, in this step, we identify identical sets of provided operations and create a set of tuples $P$ that contain the module index $j$ and the respective set of operations:

$P=\left\{\left(j,provided\text{\_}ops\left(i,j\right)\right)|i\in 0\cdots n\right\}$

This step can be augmented by defining a similarity, allowing the merging of sets of operations.

Step Four: Identifies the use of provided interfaces and creates matching required interfaces to the required modules. For this purpose, we define the function $requiring\text{\_}modules$ that returns the set of indices of all requiring modules for a given provided interface $I_p$, where $I_p$ represents the second element of a tuple of $P$:

$requiring\text{\_}modules\left(I_p\right)=\left\{i|\exists j,provided\text{\_}ops\left(i,j\right)=I_p\right\}$

After this discovery, the architecture model is enriched by adding interface declarations to modules that provide these interfaces. Similarly, each module that requires an interface of another module obtains a required interface declaration ( $requiring\text{\_}modules$) that contains a reference to the corresponding provided interface ( $provided\text{\_}ops$). Thus, the previously discovered architecture model obtains interface declarations and usage information.

As an example, the results of our interface discovery applied to UVic are presented in fig:uvic-experiment.

Software engineering aims to modularize software in a way that cohesion is high while coupling among modules is low, and the overall complexity does not increase more than the size of a software project. These direct metrics can be used to measure the maintainability and comprehensibility of large software projects [16] . Quality metrics help us to evaluate the quality of architectural designs. They provide insights into the structural quality of architectures or parts of them.

Our architecture evaluation uses complexity, coupling, and cohesion metrics based on connections between Fortran procedures grouped by files or directories. Specifically, we used two sets of metrics based on counting and information theory. Both metrics work on a coupling graph of the software system, where nodes represent procedures and edges reflect the relationships between these nodes. A relationship can be a call or multiple calls, which both result in one edge, and dataflows, which result in one or two edges, depending on their directionality. Unidirectional dataflow results in an edge representing the flow direction, and bidirectional dataflow results in two edges. Sections 0.0.1 and 0.0.2 present our architecture evaluation via counting metrics and information theory metrics, respectively.

Counting metrics compute the number of incoming and outgoing edges for each node. They work on the coupling graph introduced above. These metrics provide us with values for each procedure and cumulative values for the whole system.

Many different software quality metrics have been studied in the literature. For our study, we chose metrics that have been used in the context of software optimization previously:

These metrics have been used in [17] for software automatization.

Allen proposed information theory-based metrics for size, complexity, and cohesion [18] . These metrics are designed to provide a language-independent way of measuring the size, complexity, coupling, and cohesion of software architectures. This metric is used to represent the mental load of a software developer and can, therefore, be used to indicate architecture degradation between versions or the improvement of the software architecture.

These information theory metrics work with unidirectional graphs. The size metric computes the information content of the graph representing the system. The complexity metric is a sum over subgraphs for each node with edges, i.e., for each node, a subgraph is generated containing only those nodes of the system that are directly connected. Finally, the coupling metric operates on the subgraph only containing inter-module edges and computes the complexity of this subgraph.

The size usually grows with the number of nodes (procedures) and edges. The complexity depicts the interconnectedness of the graph. If the complexity increases faster than the size, it is an indicator of architectural degradation. Another indicator of architecture degradation is a faster growth of the coupling metric, which focuses on inter-module connections.

Since we have static and dynamic analyses available, and in the case of static analysis, we can choose between analyzing only call relationships, only dataflow relationships, or both, we have six different analysis methods, displayed in Figure 6 . The containedness relationships between the analysis methods refer to the identified coupling relationships. As an example, the analysis method combined/call is obtained by simply merging the results of the methods dynamic/call and static/call. To keep this presentation succinct, in the sequel, we mainly focus on the two analysis methods, dynamic/call and combined/both.

To explain the different kinds of software analyses techniques and their relationships, we first apply the two software quality metrics, StrCoup and LStrCoh, to our ESMs.

Figure 7 depicts our software quality metrics for the ESMs that we study in this section (namely Global Ocean cs32 × 15, Barotropic Gyre, and Global Oce Biogeo) and for UVic. For each of these four variants, we present the results of the six analysis methods from Figure 6 . Hence, we have 24 analysis results in Figure 7 . We use shapes to specify the analysis types and colors to specify the specific ESM variants. As an example, circles represent static/callanalyses, and the color red describes MITgcm cs 32 × 15. Therefore, the red circle specifies the values for the tatic/callanalysis of MITgcm cs 32 × 15, with a LStrCoh value of 4204 and a StrCoup value of 39.

Note that there are significant differences between the analysis results in Figure 6 . In particular, the coupling metrics are significantly higher for the analyses including dataflow than for the ones only considering control-flow calls: Each value for analysis including dataflow is higher than each value for each analysis not including dataflow. Further, the dynamic-only analyses metrics report much smaller values for both metrics than the analyses, which also include static analysis data.

In order to study the differences between the different analysis results, we next take a closer look at the analyses and consider the components, their size, and connections in the following subsection.

We emphasize again that the metrics given here always measure the existing structure of the software. Therefore, the differences come from applying the different analysis techniques (see Figure 6 ), not from any difference in software quality. This highlights that metrics like StrCoup or LStrCoh are performed on abstractions, like in our case, coupling graphs, and their results greatly depend on how these abstractions are obtained from the original software.

We now take a closer look at two analysis results of UVic, namely the dynamic/call analysis in Table 1 and the combined/call in Table 2 . For each component of these analyses, we list its size (i.e., the number of its units), as well as its number of independent pairs and its fan-out. The second-to-last line in each table contains the sum of the component sizes (i.e., the number of units found by the corresponding analysis), as well as the average values of the columns indPairs and fanOut—i.e., LStrCoh and StrCoup, respectively.

There are some notable differences between these analysis results. Obviously, the purely dynamic analysis ( Table 1 ) has the lowest coverage with a size of 305 units, whereas the static analysis (table omitted here) identifies a size of 439 units. The combination of both architecture models of UVic ( Table 2 ) measures 456 units, indicating that there are calls that are not detectable by the static analysis, like library calls, that are covered by the dynamic analysis. The static dataflow analysis has a higher size yet, as often data flows in both directions along calls and access to common blocks, i.e., shared global variables, are also incorporated in the analysis graph. This most complex graph is produced when combining dynamic calls, static calls, and dataflows, resulting in 477.

In general, such differences between dynamic and static analyses are not surprising: While static analysis, by design, takes the entire software into account, dynamic analysis only records information about the parts of the program used in the studied run of the system. However, the dynamic analysis allows us to identify the portions of the ESM that were actually used. Difference graphs can be generated to indicate deviations that might lead to the removal of unused code and refine interfaces.

In UVic, the normalized StrCoup values range from 0.06 to 0.07 for the call-only analyses and from 0.14 to 0.15 for the analysis, including dataflow.

Similarly, the normalized values for LStrCoh are also very close to each other: With the exception of the dynamic/call analysis (which is an outlier as it is based on a relatively small set of call data, as discussed above), the resulting values for UVic, these are between 0.0121 and 0.0129. In contrast to StrCoup, adding dataflow does not make a big difference here. The reason is that here, we count the number of independent pairs of units, so whether edges are bidirectional or uni-directional does not make a difference.

These normalizations also serve as an indicator that MITgcm’s internal structure is, in fact, better than that of UVic, as the normalized StrCoup metrics are approximately half of the corresponding value for UVic for all of our analysis techniques.

As shown in Table 3 , we computed for UVic Versions 2.6 to 2.9.2 their size and complexity utilizing the Allen metrics [18] .

As Table 3 shows, the complexity increases faster, especially from Version 2.8 to Version 2.9, indicating increasing inter-dependencies and, therefore, architecture degradation. In our interviews, the model developers referred to the code as ‘spaghetti code.’ Between Versions 2.7.5 and 2.8, we can see that the complexity increases while the lines of code decrease, indicating growing interconnections.

Out of the three MITgcm variants mentioned in Figure 7 , we focus on Global Ocean cs32 × 15 since this is the largest of the three. The results and takeaways are comparable for these three variants.

In the same way, as for UVic above, we now take a closer look at three analysis results of MITgcm Global ocean cs32 × 15, namely the dynamic/call, combinedPlainCallMap, and ombined/both analyses (see Tables 4-6 ).

As expected, the absolute values of these metrics differ significantly between the different analyses. One reason for this is that, as discussed before, the size of the models resulting from the analyses is quite different (this is apparent from the “size” column in Table 4 , Table 5 ).

To allow for more meaningful comparisons of the metric values that account for the size differences, we also consider normalized values of LStrCoh and StrCoup in the last line of each table. These values are obtained by dividing the metric values by (for StrCoup), the number of units discovered by the analysis, and (for LStrCoh) the squared number of units. We treat these two differently since StrCoup counts connections to units and, therefore, is a linear measure, while LStrCoh counts pairs of units and is therefore, quadratic.

From our results, we can also observe the above-mentioned difference between analyses including dataflow and those using only call data (comparing the analyses combined/call from Table 5 and ombined/both from Table 6 ):

While the number of detected units is pretty close (the analysis including dataflow adds a further 35 units; these are most likely Fortran common blocks), the average fan-out of the components (i.e., StrCoup) is more than doubled by adding dataflow analysis. This is because dataflow usually exists in both directions between two operations, if data flows at all (e.g., with parameters in one direction and return values in the other direction), while call relationships often only exist in one direction. Therefore, the number of directed edges is much larger for a dataflow analysis.

Note also that the normalized values of StrCoup are very close to each other: For the variant Global Ocean cs32 × 15 shown in the tables, the normalized values of StrCoup range from 0.03 to 0.07 (compared to the original ranges 2.84 to 87.32 for MITgcm). More interestingly, the thus-normalized StrCoup is 0.03 for all MITgcm analyses, including only call relationships ( Table 3 and Table 5 , and the analysis results found in the supplementary material), and is 0.06 or 0.07 for the MITgcm analyses that include dataflow relationships. Hence, the normalizations indicate that the increased coverage of the analysis methods explains some of the differences observed in the metrics. This also shows that normalization captures the difference between analyses including dataflow and those omitting it better than the original metrics.

Similarly, the normalized values for LStrCoh are also very close to each other: With the exception of the dynamic/call analysis (which is an outlier as it is based on a relatively small set of call data, as discussed above), the resulting values for MITgcm are all between 0.0025 and 0.0031. These measurements are only a 1/4 of UVic measurements, which are between 0.0121 and 0.0129, indicating that the overall architecture of MITgcm is in better shape.

In contrast to StrCoup, adding dataflow does not make a big difference here. This is because here, we count the number of independent pairs of units, so whether edges are bidirectional or unidirectional does not make a difference.

For MITgcm, we computed complexity and size with the Allen metrics and estimated the coupling degree for different variants and versions. Due to the large number of variants, we display the results as a histogram in Figure 8 . It shows that most variants range from 6000 to 7000, and more complex variants are less frequent.

We compared the size and complexity based on the Allen metrics, see Figure 9 . The data suggests that complexity grows faster than size. The fitted curve describes the relationship of the complexity to the size as logarithmic.

There is a group of outliers of 3 ESM variants between 5000 and 6500 bits of complexity that have exceptionally high values. These variants use the fizhi module that addresses atmosphere physics. This indicates that this module introduces a lot of procedures, but fewer calls than in the remaining model.

As shown in Figure 9 , the largest and most complex ESM variant is Global Ocean cs32 × 15. Thus, we picked this variant to compute the Allen metrics [18] for all versions of the ESM.

In Table 7 , we depict major versions of MITgcm, whereas in fig:mitgcm:allen, we show the measurements for all versions from checkpoint54 to checkpoint69a.

We omit the lines of code measurements (LOC), as due to the build system of MITgcm, we could not safely detect whuch parts of the code are actually used for a model variant, and we could not extract this information from the Fortran parser used to collect calls and dataflows.

As the table suggests complexity and size increase over time with some exceptions indicating major architectural changes. As MITgcm has the aim to be a modular ESM that can be used for a wide variety of experiments, such efforts are necessary to be able to be used in that manor. The ratio between complexity and size show that the ESM improves over time, but in the recent version, the ratio increased again.

In Figure 10 , we can see a sudden jump in size around checkpoint59a, which also increased the complexity, while at the same time, efforts were made to reduce the coupling of modules in MITgcm, effectively improving the architecture. Another effort to reduce coupling was made between checkpoint62f and checkpoint62g, resulting in less complex major release checkpoint63. After that, the complexity started to increase again without an impact on the overall size and with no relevant impact on the coupling. This shows that functionality inside modules was added or improved. Only in more recent versions did new functionality arrive and cause a large increase in coupling and complexity.