The electron microscope (EM) instrument, initially developed by Ernst Ruska in the early nineteen thirties [1] , became a routine scientific instrument during the nineteen fifties and sixties. With the gradual development of appropriate specimen-preparation techniques, it proved an invaluable tool for visualizing biological structures. For example, ribosomes, originally named “Palade particles”, were first discovered in the nineteen fifties in electron microscopical images [2] . In the nineteen sixties and seventies, the early days of single-particle electron microscopy, the main specimen preparation approach used for investigating the structure of biological macromolecules was the negative stain technique in which the samples were contrasted with heavy metal salts like uranyl acetate [3] [4] . In those days, the standard way of interpreting the structures was to come up with an intuitively acceptable three-dimensional arrangement of subunits that would fit with the observed (noisy) molecular images.

Around 1970, in a number of ground-breaking publications, the idea was introduced by the group of Aaron Klug in Cambridge of using images of highly symmetric protein assemblies such as helical assemblies or icosahedral viral capsids [5] [6] to actually calculate the three-dimensional (3D) structures of these assemblies. The images of these highly symmetric assemblies can often be averaged in three dimensions without extensive pre- processing of the associated original images. Averaging the many unit cells of a two-dimensional crystal, in combination with tilting of the sample holder gave the very first 3D structure of a membrane protein [7] . Electron tomography of single particles had been proposed by Hoppe and his co-workers [8] , however, due to the radiation-sensitivity of biological macromolecules to electrons, it is not feasible to expose a biological molecule to the dose required to reveal the 3D structure from one hundred different projection images of the same molecule.

For other types of complexes, no methods were available for investigating their 3D structures. The vast majority of the publications of those days thus were based on the visual recognition of specific molecular views and their interpretation in terms of the three-dimensional of the macromolecules. For example, a large literature body existed on the 3D structure of the ribosome-based antibody labelling experiments [9] [10] . The problem with the visual interpretation of two-dimensional molecular images was obviously the lack of objectivity and reproducibility in the analyses. Based on essentially the same data, entirely different 3D models for the ribosome had been proposed [9] . There clearly was a need for more objective methods for dealing with two-dimensional images of three-dimensional biological macromolecules.

Against this background, one of us (MvH), then at the university of Groningen, and Joachim Frank, in Albany NY, started a joint project in 1979 to allow for objectively recognizing specific molecular views in electron micrographs of a negative stain preparation (see Appendix). This would allow one to average similar images prior to further processing and interpretation. Averaging is necessary in single-particle processing in order to improve the very poor signal-to-noise ratios (SNR) of direct, raw electron images. Averaging similar images from a mixed population of images, however, only makes sense if that averaging is based on a coherent strategy for deciding which images are sufficiently similar. We need good similarity measures between images such as correlation values or distance criteria for the purpose. Upon a suggestion of Jean-Pierre Bretaudière (see Appendix), the first application of multivariate statistical analysis (MSA) emerged in the form of correspondence analysis [11] [12] . Correspondence Analysis (CA) is based on chi-square distances [13] - [15] , which are excellent for positive (histogram) data. In retrospect, other distances are more appropriate for use in transmission electron microscopy (see below) but let us not jump ahead.

The above MSA eigenvector-eigenvalue data compression was still incomplete without the introduction of automatic “unsupervised” classification in the resulting compressed eigenvector space [16] [17] . It is in the classification phase where the actual decision-making takes place of which images are to be averaged in order to produce the desired low-noise averages (we will discuss this in extensor below). (Note that this first EM classification paper already contains various illustrations of what are now known as “manifolds”, representing the continuous structural orientations in hyperspace [16] ). Since this introduction of unsupervised classification in the EM field, a plethora of other classification algorithms have been introduced that rather fall in the “supervised” classification regime. These classification algorithms are typically an iterative mix of alignments with respect randomly selected references images followed by the summing of similar images. Often these algorithms are referred to as “reference free” 2D classification schemes, although they invariably do use references for cross- correlation based alignments, albeit that they are randomly selected from the input images. Examples of this family of ad-hoc algorithms for the simultaneous alignment and classification of 2D images or of 3D reconstruction volumes include: [18] [19] . We will leave these out of our or considerations because they do not fit the eigenvector formalism of this paper.

A most fundamental specimen-preparation development strongly influenced the field of single-particle electron microscopy: In the early nineteen eighties the group around Jacques Dubochet at the EMBL in Heidelberg brought the “vitreous-ice” embedding technique for biological macromolecules to maturity [20] . Already some 25 years earlier, the Venezuelan Humberto Fernández-Morán suggested the idea of rapid freezing biological samples in liquid helium in order to keep them hydrated [21] . Marc Adrian and Jacques Dubochet brought the rapid-freezing technique to maturity by freezing samples in liquid ethane and propane, cooled au-bain-marie in liquid nitrogen [20] . This is the practical approach that is now used routinely (for a review see [22] ). The vitreous-ice specimen preparation technique represented a major breakthrough in structural preservation of biological specimens in the harsh vacuum of the electron microscope. With the classical dry negative-stain preparation technique-the earlier standard-the molecules tended to distort and lie in just a few preferred orientations on the support film. The vitreous-ice, or better, “vitreous-water” technique greatly improved the applicability of quantitative single particle approaches in electron microscopy.

The instrumental developments in cryo-EM over the last decades have been substantial. The new generation of electron microscopes-optimized for cryo-EM-are computer controlled and equipped with advanced digital cameras (see, for example: [23] ). These developments enormously facilitated the automatic collection of large cryo-EM data sets (see, for example [24] ). A dramatic improvement of the instrumentation was the introduction of direct electron detectors, which represent an efficiency increase of about a factor four with respect to the previous generation of digital cameras for electron microscopy [25] [26] . This development also allowed for data collection in “movie mode”: Rather than just collecting a single image of a sample, one collects many frames (say 5 to 50 frames) which can later be aligned and averaged to a single image [27] [28] . The sample is moving continuously over many Ångstrom during the electron exposure which has the effect of a low-pass filter on the single averaged image; “frame alignment” can largely compensates for this effect. Single particle cryo-EM thus recently went through a true “Resolution revolution” [29] .

These newest developments facilitate the automatic collection of large dataset such as are required to bring subtle structural/functional differences existing within the sample to statistical significance. The introduction of movie-mode data collection by itself already increased the size of the acquired data set by an order of magnitude. Cryo-EM datasets already often exceed 5000 movies of each 10 frames, or a total of 50,000 frames of each 4096 × 4096 pixels. This corresponds to 1.7 Tb when acquired at 16 bits/pixels (or 3.4 Tb for 32 bit/pixel). From this data one would be able to extract ~1,000,000 molecular images of each, say, 300 × 300 pixels. These 1,000,000 molecular images represent 100,000 molecular movies, of each 10 frames. The MSA approaches are excellent tools to reveal the information hidden in our huge, multi-Terabyte datasets.

First, One image is simply a “measurement” that can be seen as a collection of numbers, each number repre- senting the density in one pixel of the image. For example, let us assume the images we are interested in are of the size 300 × 300 pixels. This image thus consists of 300 × 300 = 90,000 density values, starting at the top left with the density of pixel (1, 1) and ending at the lower right with density of pixel (300, 300). The fact that an image is intrinsically two-dimensional is not relevant for what follows. What is relevant is that we are trying to make sense out of a large number of comparable measurements, say 200,000 images, all of the same size, with pixels arranged in the same order. Each of these measurements can be represented formally by a vector of numbers F(a), where a is an index running over all pixels that the image contains (90,000 in our case). A vector of numbers can be seen as a line from the origin, ending at one specific point of a multi-dimensional space known as hyperspace.

There is no “magic” in hyperspace; it is merely a convenient way to represent a measurement. A set of 200,000 images translates to a set of 200,000 points in hyperspace (with 90,000 dimensions, in our case). When trying to make sense of these 200,000 different molecular images, we like to think in terms of the distances between these points. The collection of our 200,000 points in hyperspace is called a data cloud. Similar images are close together in the data cloud and are separated by only a small distance, or, equivalently, have a high degree of correlation as will be discussed in the next chapter. This abstract representation of sets of measurements is applicable to any form of multidimensional data including one-dimensional (1D) spectra, two-dimensional (2D) images, or even three-dimensional (3D) volumes, since all of them will be represented simply by a vector of numbers F(a).

The entire raw data set, in our example, consists of 200,000 images of each 90,000 pixels or a total of 18 × 10⁹ pixel density measurements. In the hyperspace representation, this translates to a cloud of 200,000 points in a 90,000 dimensional hyperspace, again consisting of 18 × 10⁹ co-ordinates. Each co-ordinate corresponds to one of the original pixel densities, thus the hyperspace representation does not change the data in any way. This type of representation is illustrated in Figure 1 for a data set simplified to the extreme, where each image consists of just two pixels.

The basic idea of the MSA approach is to optimize the orthogonal co-ordinate system of the hyperspace to best fit the shape of the cloud. We wish to rotate (and possibly shift) the co-ordinate system, such that the first axis of the rotated co-ordinate system will correspond to the direction of the largest elongation of the data cloud. In the simplistic (two-pixel images) example of Figure 1, the largest elongation of the cloud is associated with the average density in the two pixels increasing together. That main direction points from the lower left of the illustration (both pixels have low density) to the top right (both pixels high density). The remaining direction is perpendicular to the first one (also indicated in the illustration) but that direction describes only small modulations with respect to the main trend of the data set and may be ignored. The power of the MSA approach lies in this data reduction. It allows us to then concentrate on the most important trends and variations found in a complex data set and ignore all the other sources of fluctuations (which in EM usually is just noise). We thus enormously reduce the total amount of data into just a few main “principal” components, which show the most significant variations of the data cloud.

Concentrating on the main direction of variations within the data, in the example of Figure 1, reduces the problem from a two-dimensional one to just a one-dimensional problem. This reduction of dimensionality can take dramatic proportions with real EM data sets. In the case of a data set of two hundred thousand images of 300 × 300 pixels, typically some 50 - 200 orthogonal directions suffice to describe the most significant (largest) sources of variations within the data. Each image is then no longer described by the 90,000 density values of its pixels, but rather by just its 50 - 200 co-ordinates with respect to those main directions of variations. This represents a reduction in the dimensionality of the data by more than three orders of magnitude. After this data reduction, it becomes feasible to perform exhaustive comparisons between all images in the data set at a reasonable cost. The main orthogonal directions of variations within the data are known as “eigenvectors”; the variance that each eigenvector describes is known as the corresponding eigenvalue. Because each eigenvector corresponds to a point in hyperspace these, in our case, have the character of “image” and are therefore usually referred to as “eigenimages”.

Multivariate Statistical Analysis is all about comparing large sets of measurements and the first question to be resolved is how to compare them. What measure of similarity one would want to use? The concepts of distances and correlations between measurements are closely related, as we will see below. Different distance and associated correlation criteria are possible depending on the metric one chooses to work with. We will start with the simplest and most widely used metric: The classical Euclidean metric.

We have thus far considered an image (or rather any measurement) as a vector named F(a) or as a vector named G(a). We want to study large data sets of n different images (say: 200,000 images), each containing p pixels (say 90,000 pixels). For the description of such data sets we use the much more compact matrix notation. For completeness, we will repeat some basic matrix formulation allowing the first time reader to remain within the notation used here.

The mathematics of the PCA eigenvector eigenvalue procedures have been described in various places (for example in [14] [15] [33] ). We here try to follow what we consider the best of earlier presentations with a bit of a further personal twist. We want to find a unit vector that best describes the main direction of elongation of the data cloud. “Best” here means finding a direction which best describes the variance of the data cloud.

We have introduced various distances and correlation measures earlier, but in discussing the MSA approaches we have so far only considered conventional Euclidean metrics.

The algorithm we use for finding the main eigenvectors and eigenvalues of the data cloud is itself illustrative for the whole data compression operation. The IMAGIC “MSA” program, originally written by one of us (MvH) in the early 1980s, is optimised for efficiently finding the predominant eigenvectors/eigenvalues of extremely large sets of images. Here we give a simplified version of the underlying mathematics. Excluded from the mathematics presented here are the “metric” matrices N and M for didactical reasons. The basic principle of the MSA algorithm is the old and relatively simple “power” procedure (cf. [35] ; also discussed in Wikipedia under “eigenvector power iteration”). In this traditional approach one multiplies a randomly chosen vector r₁, through the symmetric variance co-variance matrix A, which will yield a new vector:

This resulting vector is then (after normalisation) successively multiplied through the matrix A again:

and that procedure is then repeated iteratively. The resulting vector will gradually converge towards the first (largest) eigenvector u₁ of the system, for which, per definition, the following equation holds:

Why do these iterative multiplications necessarily iterate towards the largest eigenvector of the system? The reason is that the eigenvectors “u” form a basis of the n-dimensional data space and that means that our random vector r₁ can be expressed as a linear combination of the eigenvectors:

The iterative multiplication through the variance-covariance matrix A will yield for r₁ after k iterations (using Equation (58) repeatedly):

or:

Because λ₁ is the predominant eigenvalue, the contributions of the other terms will rapidly vanish (; i > 1), and these iterations will thus make r₁ rapidly converge towards the main eigenvector u_1. The variance co-variance matrix A is normally calculated as the matrix multiplication of the data matrix X and it's transposed, X^T:

As was mentioned above, the data matrix X contains, as its first row, all of the pixels of image 1; its general i^th row contains all the pixels of image i. The MSA algorithm operates by multiplying a set of randomly generated eigenvectors (because of the nature of the data also called eigenimages) r₁, r₂, etc., through the data matrix U and its transposed U' respectively. The variance-covariance matrix A_p is thus never calculated explicitly since that operation is already too expensive in terms of its massive computational burden. The MSA algorithm does not use only one random starting vector for the iterations, but rather uses the full set of q eigenimages desired and multiplies that iteratively through the data matrix X, similar to what was suggested by [36] .

In detail the MSA algorithm works as follows (Figure 3). The eigenvector matrix U_q is first filled with random numbers which are then orthonormalised (normalised and made orthogonal to each other). The typical number of eigenimages used depends fully on the complexity of the problem at hand but typically is 10 - 100 and they are symbolised by a set of two “eigenimages” in the illustration (top of Figure 3). Then, the inner product between these images and the n images of the input data set is calculated. This calculation leads to coefficient vectors of length n as indicated in the left-hand side of Figure 3. The next step is then to calculate weighted sums over the input image stack, using the different coefficient vectors as the weights for the summing. A new set of eigenimage approximations is thus generated as shown in the lower part of Figure 3. New approximations are generated from this set by orthonormalization and over-relaxation with respect to the previous

set. The algorithm converges rapidly (typically within 30 - 50 iterations) to the most important eigenimages of the data set.

An important property of this algorithm is its efficiency for large numbers of images n: its computational requirements scale proportionally to n∙p, assuming the number of active pixels in each image to be p. Most eigenvector-eigenvalue algorithms require the variance-covariance matrix as input. The calculation of the variance-covariance matrix, however, is itself a computationally expensive algorithm requiring computational resources almost proportional to n³. (This number is actually: Min (n²p, np²)). The MSA program produces both the eigenimages and the associated eigenpixel-vectors in the conjugate data space as described in [33] . One of the intuitive charms of this fast disk-based eigenvector-eigenvalue algorithm is that it literally sifts through the image information while finding the main eigenimages of the data set. The programs have been used routinely for more than 30 years, on a large number data sets consisting of up to ~1,000,000 individual images.

In spite of its high efficiency and perfect scaling with the continuously expanding sizes of the data sets, compared to most conventional eigenvector-eigenvalue algorithms, the single-CPU version of the algorithm had become a serious bottleneck for the processing of large cryo-EM data sets. The parallelisation of the MSA algorithm had thus moved to the top of our priority list. We have considered various parallelisation schemes including the one depicted in Figure 4; this mapping of the computational problem onto a cluster of computers was indeed found to be efficient.

As was discussed before, Equation (46a) (see also the legend of Figure 3) is the full matrix equivalent of:

; (62)

In which, u_j is one single eigenvector and λ_j the corresponding eigenvalue. Having stated this, a direct approach for parallelisation of the calculation of Equation (46a) could be the exploiting of the independence of calculations of all u_j since each eigenvector depends only on one eigenvector u_j. Although it is a clear possibility and is straightforward to implement, this scheme does not represent an effective approach. Since there may typically be more processors available than the number of eigenvectors needed for representing the dataset, this approach would lead to a waste of processing time with many processors runningidle. Moreover, all processors would need access to the full image matrix X. This approach is thus likely to create severe I/O bottlenecks. To analyse another possible parallelisation schemes, let us write Equation (62) as:

where u_ij is element i of eigenvector u_j, x_ij is pixel j of image i, and n_s and m_t are the diagonal elements of the N and M metrics, respectively. This equation can be expanded into the form:

The equation above suggests a possible parallelisation scheme over the image dataset instead of the eigenvectors (Figure 4). Each processor calculates a partial solution for based on a subset of k images. After the complete calculation of U', the further steps of the algorithm are computed in a non-parallel way. The algorithm thus consists of a parallel and a non-parallel part.

The implementation of the algorithm can be structured in a master/slave architecture, where the master is responsible for all non-parallel tasks and for summing the partial results of each node. During each iteration of the algorithm, each node needs to read data from the image matrix X thus potentially creating a network bottleneck. However, since each node in this parallelisation scheme needs only to access a part of the X matrix, distributing the partial datasets of input data over the local disks efficiently, parallelises the I/O operations.

What we wish to achieve with the MSA approaches is to concentrate on the intrinsic information of large noisy data sets and make that more accessible. The determination of the eigenvectors (eigenimages) by itself does not change or reduce the total image information in any way. It is merely a rotation of the co-ordinate system in a special direction, such that the first eigenvector covers most of the variance in the data cloud, the second, orthogonal to the first, covers most of the remaining variance of the data cloud, not covered by the first eigenvector, etc. If one would find all the eigenvectors of the data cloud (all the eigenvectors of the variance-covariance matrix; all p or n of them, whichever is the smaller number) there would be no loss of information at all. Of course, even without any loss of information the most important information is associated with the first eigenvectors/eigenimages that describe the strongest variations (in terms of variance) within the data set.

When we decide to consider only the first so-many eigenvectors and thus decide to ignore all higher eigenvectors of the system, we make the “political” decision between what is signal and what is noise. Here the decision that the percentage of the overall variance of the data set (the “trace”) covered by the main eigenvectors, suffices for our desired level of understanding of the dataset. An often spectacular level of data compression is thus achieved. The problem remains of how to best exploit that information in the compressed data space. Having concentrated on the mathematics and algorithmic aspects of the eigenvector-eigenvalue aspect of the MSA approach, we here will just give a relatively short account of this most fascinating aspect of the methodology.

The first EM data sets that were ever subjected to the MSA approach were simple negative-stain contrasted specimens where the differences were directly visible in the micrographs. The data sets were chosen carefully for being simple in order for the problem to be solvable with the limited computing resources available in those days. The very first data set that was ever analysed by the MSA approach was an artificial mixture dataset from two different species of hemoglobins. The first analyses were performed with the “AFC” program developed by Jean Pierre Bretaudière from a program originally written by JP Benzécri (see Appendix for details). The 16 molecular images of the giant annelid hemoglobin of Oenone fulgida visibly had extra density in the centre compared to the 16 molecular images of well-known giant hemoglobin of the common earthworm Lumbricus terrestris which rather had a “hole” in the centre of its double-ring geometry (see Appendix). The separation of the two species was so obvious that one immediately moved on to a single data set with real internal variations (the example below).

The first real data set with genuine variations among the molecular images was a preparation Limulus polyphemus hemocyanin half-molecules, each consisting of four hexamers produced by controlled dissociation of the full molecule. The data sets were by today's standards extremely small with only 46 images in this case. Each 64 × 64 image was then low-pass filtered and coarsened to a mere 32 × 32 pixel format. Nevertheless the fact that the image information could be compressed logically to points on a two-dimensional plane (Figure 5) was a tremendous breakthrough. The logical grouping of those points on this factorial “map” immediately suggested which images to average together in order to suppress the noise present in the individual images. Which images were grouped together, however, was a direct human decision, in spite of the first line of the abstract stating: “We have developed a new technique of analysis that allows automatic classification of molecule images according to subtle differences”.

In those early days in computing, the address space of the computers was minimal by today’s standard. The 16-bit PDP-11/45 computer on which the work was performed would only address 2¹⁶ bytes = 64 Kbytes of memory. It was thus difficult to fit any serious computational problem in the central memory of that generation of computers! The critical memory requirement for the eigenvector-eigenvalue calculations was the “core” needed for the square variance-covariance matrix which, practically limiting the number of “particles” to around 50 at most. In the IMAGIC implementation of Bretaudière’s AFC program (see Appendix) the limit was slightly over 100 molecular images (see the Limulus polyphemus hemocyanin double-hexamer analysis in [39] ). That larger number of particles was purely due to the fact that the NORSK DATA Nord-10 computer addressed 16-bit word rather than the 8-bit words of the PDP-11/45, a doubling of the usable memory. With the computer hardware limiting the complexity of problems that could be reasonably studied, the limitations of being able to only look at problems that were intrinsically two-dimensional, was not immediately felt. It was more frustrating that the averaging was over classes that still were a subjective choice of the user after the elegant data compression.

A new generation of 32-bit computers―with a much larger address space―came on the market (such as the Digital Equipment VAX-780) allowing one to handle a much larger data sets with a significantly higher level of complexity. There were some efforts to map more complicated cases nonlinearly onto just a 2D plane in order to visualise the underlying structure. Although such approaches may relieve the problem somewhat, what was really needed was a more abstract approach in which all information in the compressed data space is structured automatically. The development of automatic classification procedures operating on a, say, 60-dimensional compressed data space became an absolute necessity.

To eliminate the subjectivity of visual classification (which is simply impossible when the number of factors to be taken into account is larger than two or three), automatic classification schemes were introduced in EM. These schemes emerged in the general statistical literature in the 1960s and 1970s, and were first used in electron microscopy in the early 1980s. There are many algorithms available each with their advantages and disadvantages. The three listed below are important in single particle cryo-EM.

Another example of the use of MSA is the unbiased analysis of the main symmetry properties of a macromolecular complex. Earlier symmetry analysis approaches were based on finding the predominant rotational symmetry components of one single image at a time [49] . That single image could then be used to align all particles of the data set. However, when aligning a whole set of images with respect to one single image with a strong, say, 6-fold symmetry component, the 6-fold symmetry property will be implicitly imposed upon the whole data set by the reference-bias effect [50] [51] during the alignment procedure. When an average is then obtained from the aligned images, the 6-fold symmetry component becomes overwhelmingly present, but that serves only as a demonstration of the reference bias effect, and certainly not as a proof of the real symmetry of the complex being studied. Various papers have appeared in the literature in which this “self-fulfilling prophecy” approach was used for determining the “true” symmetry of an oligomeric biological structure.

A methodologically clean approach for determining the strongest symmetry component in a data was proposed, which entirely avoids the symmetry bias resulting from any explicit or implicit rotational alignment of the molecular images [52] . In this approach, only a translational alignment relative to a rotationally symmetric “blob” is performed for centring particles. The rotational orientation of all molecules remains arbitrary, and will thus often be the main source of variation among the images of the data set. MSA eigenvector analysis will then find those main eigenimages of the system and these reflect the symmetry properties of data elegantly. The particles used here (Figure 6) resulted from an automatic particle selection over a stack of micrographs of the hemoglobin of the common earthworm Lumbricus terrestris. We use these data as a high-contrast general testing standard [53] . The particle picking program was used in cross-correlation mode with a rotationally-symmetric reference/template image (see insert in Figure 6). This particle-particle picking procedure yields centred particles equivalent to the translational alignment with a rotationally symmetric average used in the original paper [52] .

The total of some 7300 particles were submitted to MSA eigenvector analysis. The first eigenvector in this analysis (Figure 7) is almost identical to the reference image used to pick the particles. The two next eigenimages are essentially two identical 6-fold symmetrical images rotated by a small angle with respect to each other. A six-fold symmetrical structure repeats itself after a rotation over 60˚. These two eigenimages 2 and 3, are rotated by exactly 15˚ degrees with respect to each other. The eigenimages 2 and 3 consist of sine waves along the tangential direction of the images and along each circle around the centre of the eigenimage we will see 6 full periods of that sine wave. One period of the sine wave corresponds to a 60˚ arc on the circle. The 15˚ rotation between the two eigenimages mean that the sine waves at each radius of the two images are π/2 out of phase with each other, like a sine and a cosine function. Note also (from the eigenvalue plot in Figure 7) that they have an associated eigenvalue of more than twice the magnitude of all subsequent eigenvalues.

Alignments of the images within the data matrix X change the MSA analysis in fundamental ways. Interestingly however, alignments of the individual images do not change the total variance of the data set. A rotation of an image merely shifts around the pixel densities within a row of the data matrix X; the total variance of the measurement, however, does not change. (This is perfectly true as long as no non-zero pixels are rotated or shifted out of the part of the image that is active during the MSA analysis). The variance of each image is the corresponding diagonal element of the variance-covariance matrix A and hence these diagonal elements remain unchanged.

The total variance of the data set (the trace of A) is the sum of the diagonal elements of matrix A (Equation (21a)). What does change during the alignment procedures is that certain images within the data matrix will become better aligned to each other and will sense a higher co-variance among themselves (specific off-diagonal elements of matrix “A” will increase).

We will here apply an interesting and illustrative form of alignment, namely: “alignment by MSA”, (similar to the Alignment By Classification “ABC” [46] [52] ) to the worm hemoglobin data set introduced above (Figure 6). Having noticed that eigenvectors 2 - 7, all are associated with 6-fold symmetry or the harmonics thereof, we choose to align the (already centred) data set-in a rotational sense only-with respect to eigenimage 2 (Figure 8). The rotation alignment was restricted in range of −30˚ to +30˚ in order to leave the 6-fold-symmetry character of the data set undisturbed. Eigenimage 2 is pasted in the top-left corner of the illustration, whereas all the other images in this figure are the rotationally aligned versions of the centred particle of the original data set (Figure 6).

We have here aligned the full (centred) data set rotationally with respect to one of the two main eigenimages associated with the 6-fold symmetry property of this hemoglobin. In doing so, we have concentrated a much larger percentage of the total variance of the data set into the first few eigenvectors of the system as is seen from the total sum of the first 10 eigenvalues before (4.3%; Figure 7) and after the rotation-only alignment (6.6%; Figure 9). As was mentioned above, an alignment does not change the variance of each individual image and thus does not change the total variance of the data set.

Concentrating the information (variance) into the first few eigenimages means that more eigenvectors will become statistically significant above the background random noise level. With the relatively small data sets from the past, alignments were an absolute necessity for obtaining statistically significant results. Today, however,

and especially with the powerful new parallel MSA algorithms, we can afford to largely increase the overall size of the data set. The rationale for using larger data sets is simple: using ten times more images in a data set means using ten times more information! In doing so, we can achieve statistical significance without the risk prejudicing the data sets by introducing reference bias [50] [51] . This increased level of objectivity is especially important when trying to find objectively find different conformations in a heterogeneous population of molecular images.

Mixed populations of molecular images were originally the curse of single-particle electron microscopy. Originally, one would try to work with an as homogeneous as possible population of macromolecules. The basic assumption for single particle cryo-EM was that all molecules were the same, apart from the fact that each molecule could exhibit a different orientation and be in a different position in the sample. When the data were heterogeneous or flexible, and that heterogeneity is not accounted for, that results in a deterioration of the quality of the final 3D reconstruction since one will then be averaging “cows” and “horses” into the same 3D map. When, on the other hand, one knows how to separate the molecular images into structurally homogeneous subgroups, a whole new world of possibilities opens. One can then study a mixture of closely related structures and from that commence to understand the biological reasons for the distinct 3D structures found in a sample. The name “4D” cryo-EM has been used in this context.

One of the first examples of the challenging possibilities came from a cryo-EM sample of the E. coli 70s ribosome, in complex with Release Factor 3 [54] . It was found the structure simply did not converge towards a high-resolution structure no matter how much effort was invested. However, once the data set was allowed to refine (by multi-reference alignment) towards different 3D reconstructions simultaneously, the sample turned out to consist of a mixture of ribosomes in two different conformational states. The ribosomal complexes were found to co-exist in the pre-translocation and post-translocational states of the 70S ribosome. The “RF3” was also found in two different conformations depending on the conformation of the 70S ribosome itself. Once the data set had been separated into conformational subsets, by local MSA of specific projection images, each subset of images led to a separate 3D structure exhibiting a much higher resolution than the resolution on the overall 3D reconstruction. A further example, where size variations among the particles was found to be a limiting factor, is given by [55] .

Another possibility to study 3D structural variation is to generate a large number of 3D reconstructions and to study those by means of MSA the way 2D molecular images were studied above [56] . The IMAGIC system has been optimised to manage 3D volumes with the same ease and transparency as the system handles sets of 2D images [46] . This has greatly facilitated handling these huge intricate datasets.

A quality criterion for the 4D MSA types of analysis is the resolution achieved in the subset 3D reconstructions. As usual the resolution is quantified by Fourier Shell Correlation (“FSC”, [57] [58] ). There are two conflicting issues here. It is generally true that larger datasets lead to higher resolution results than smaller datasets. An increase of resolution, while decreasing the number of raw images entering in each specific 3D reconstruction, is an important indication for the population of images indeed to belong to different structural subsets. The separation of large data sets into structurally homogeneous subsets must be seen as a major challenge in current biology and this area is in constant flow (see below). We do not want to go into much detail here but rather point the reader to a general review [59] .

The main goal of the biological sciences is to make sense of the massive and very noisy “data” in terms of elucidating the underlying general principles. No better example than the data sets collected in cryo-EM which are probably among the largest and noisiest currently being collected in the biomedical sciences. Over the past three decades, multivariate statistical analysis approaches have been very successful in helping us sort out complex EM data sets in many different ways, and examples have been discussed such as a first “manifold” separation into two functional states [54] . However, we have not yet seen the full potential of the MSA approaches exploited due the costs of the computing effort. This high cost has hampered the use and the development of potentially very useful further MSA approaches such as the “alignment by MSA” discussed above.

How can that be that the computational effort was simply not affordable? After all, one of the most dramatic developments in science over the past decades, has been the increase in speed in general computing. In spite of massive computing resources becoming available, however, we have seen an even more rapid increase in the size and complexity of the EM data sets. The size of the data sets have been limited not by what one would wish to analyse, but rather by what was practical to analyse. The high-resolution scanning of (analogue) micrographs into digital form was a tedious activity taking up to hours of labour for a single micrograph. With automatic 24/7 data collection directly on the microscope―using direct electron detector 4 k × 4 k cameras [25] [26] ―the primary data-collection bottleneck has been removed. Therefore the speed of the MSA eigenvector computations had become the new bottleneck limiting its use in cryo-EM. The recent parallelisation of MSA programs (operating in the context of the IMAGIC software system [46] [60] ) opens a whole range of new possibilities by speeding up the process by orders of magnitude, exploiting the power of hundreds of CPUs simultaneously.

There are other reasons why the MSA have not always received the appropriate attention, including ignorance. The MSA approaches are not the simplest of techniques to understand and to get acquainted to for the potential biological user. The learning curve is steep and users tend to avoid investing the time necessary to understand the approach. More seriously, however, is that many software systems in use in cryo-EM do not have MSA options, and those that have typically apply standard libraries for performing the analysis. These standard eigenvector-eigenvalue routines normally calculate all eigenvectors and eigenvalues of the system, and those are algorithms that scale proportional to N², implying that the analysis of data sets of more than a few thousand elements will easily exceed the memory/computing capacity of any modern computer. With the poor scaling of the algorithms it can be very difficult to analyse large data sets, a fact that is also not appreciated in the general statistical literature where data sets are small by cryo-EM standards. Even in the special EM literature this issue is not always appreciated. (See, for example, paragraph 2.5 of [37] where the reasoning is based on the availability of the variance-covariance matrix A, which is already too expensive to calculate as was discussed above). Therefore, in times where datasets of hundreds of thousands or millions of images are the rule, one sees MSA applications often limited to just a few thousand images.

Another issue that has largely been ignored in the literature is that the original c²-metric of correspondence analysis is inappropriate for electron-microscopical phase-contrast data. Historically it happened to be the first MSA technique used because it was available (see Appendix). However, the c²-metric correspondence analysis uses is designed for positive data (counting occurrences in histograms). To make the EM phase-contrast data positive one could threshold the image data to positive values only. This thresholding, however, is not justifiable for general signals and―at the very least―leads to wasting of half of the information for zero-average-density measurements. Alternatively, the negative values can be rendered positive by adding a constant to the data. This too has far-reaching consequences. The strong negative densities will end up as small positive densities and will have very little contribution to the total variance of the data set, whereas the large positive values will become very large positive values with a disproportionally strong contribution to the total variance. For high-resolution EM work, however, the c²-metric must be considered obsolete since the publication of [33] . The metrics that are permissible for general signal processing are the Euclidean distance of principal components analysis and the modulation distance of modulation analysis discussed above.

In recent years, another multivariate statistical approach was introduced to single particle cryo-EM known as the “maximum likelihood” [61] , an “explanatory factor analysis” technique. This approach had a significant number of successes in refining the structure of large complexes [29] [62] . A discussion of the similarities and differences of the two general approaches (“exploratory factor analysis” versus “principal component analysis”) can be found in Wikipedia from which we quote: “In factor analysis, the researcher makes the assumption that an underlying causal model exists, whereas PCA is simply a variable reduction technique”. And: “If the factor model is incorrectly formulated or the assumptions are not met, then factor analysis will give erroneous results. Factor analysis has been used successfully where adequate understanding of the system permits good initial model formulations. Principal component analysis employs a mathematical transformation to the original data with no assumptions about the form of the covariance matrix.” It is still early days in single-particle cryo-EM and in depth comparisons will be performed in the future. The maximum likelihood approaches are typically used in conjunction with reference-based alignments or projection-matching approaches which, are not reference free and are thus in principle are prone to reference bias. The original MSA approaches were preceded by multi-reference alignments [18] [30] which themselves are also not reference free. However, the Euler orientation assignments were performed separately after a separate unsupervised classification which makes the approach less prone to reference bias. The ABC approaches discussed above [46] [52] introduce a new level of reference independence.

The issue of reference bias is of fundamental importance in the cryo-EM field. There has been a much publicised case where two different 3D structure were presented by Mao & Sodrosky and co-workers [63] [64] which were seriously criticized in the field [65] - [67] . By using “supervised classification”, that is, looking for specific molecular projections in noisy micrographs, these authors found what they were looking for to begin with. Their critics accused the authors of looking for “Einstein in random noise” the classical reference-bias experiment in cryo-EM. Interestingly, the unbiased ABC approach (unsupervised classification; no reference images) was used in [65] to refute the results of those authors. Because our eigenvector-eigenvalue data compression approach is simply a “variable reduction technique” is perfectly suited for this type of neutral, unbiased analysis.

New cryo-EM developments allowing the separation of subsets of 3D structures are emerging that have the potential to change the field of structural biology. Atomic resolution structures (~3Å) had hitherto been elucidated mainly by X-ray crystallography, where the biological molecules are confined to the rigidity of a crystal. In X-ray crystallography one collects diffraction amplitudes data that per definition has been averaged over all unit cells of the crystal. For a better understanding of a biological process, however, it is essential to see the sequence of conformational changes and interactions a molecule or complex undergoes during its functional cycle. A sequence of 3D structures or “4D data analysis” is required for this level of understanding. We want to separate the different manifolds in hyperspace, representing the different states the complexes we study can assume. Single-particle cryo-EM provides a direct window into the solution, revealing different views of different complexes, in different structural/functional states [54] [68] [69] . The challenge is to extract all information from the noisy molecular images and to bring the existing structural differences to statistical significance. Revealing the full biological complexity of the data produced by 4D cryo-EM at atomic resolution remains one of the major challenges in modern structural biology. The MSA approaches discussed in this paper are among the best tools for creating a clear picture of what is happening in the solution.

New developments in MSA eigenvector-eigenvalue data-compression approaches are expanding the possibilities of single-particle cryo-EM. Decades after its first introduction to electron microscopy, MSA approaches are more alive than ever and they are being tailored for many new tasks. With the orders-of-magnitude speed increase achieved by the MPI parallelisation of the algorithms, huge datasets of Terabytes in size can processed. A rapid expansion of the multivariate statistical techniques has taken place in single-particle cryogenic electron microscopy. One of the most challenging uses of the parallel MSA algorithm is the separation of heterogeneous data sets into different 3D structures associated with the different functional states of the macromolecular complexes we seek to understand. The different functional states of the molecules we study may be better visualised and understood by assessing the different manifolds one finds in the eigenvector compressed dataspace. Thanks to the maturation and proliferation of multivariate statistical data analysis approaches, single-particle electron microscopy techniques are now one of the preferred instruments of structural biology.

First let us again point to the first MSA ideas that were inspired by Jean-Pierre Bretaudière (see Appendix). This document reflects decades of development and experience with the MSA techniques discussed. During this period many individuals contributed to the various MSA projects (and not just the current authors) including: George Harauz, Lisa Borland, Ralf Schmidt, Elena Orlova, Holger Stark, HansPeter Winkler, Manfred Kastowsky, Bruno Klaholz, Tim Balance, Charlotte Linnemayr-Seer, and Pavel Afanasiev. Countless other scientists who used the programs gave us constructive feedback, reported bugs or inconsistencies, and thus forced us to streamline our programs and ideas on MSA methodologies. We thank Katie Melua for inspiring the Beijing/Cambridge bicycle statistics example. Our research was financed in part by grants from: EU/NOE (grant No. NOE-PE0748), from the Dutch Ministry of Economic Affairs (Cyttron project No. BIBCR_PX0948; Cyttron II FES-0908); the BBSRC (Grant: BB/G015236/1); the Brazilian science foundations: CNPq (Grants CNPq- 152746/2012-9 and CNPq-400796/2012-0), and the InstitutoNacional de C, T&I emMateriais Complexos Funcionais (INOMAT).

Marin van Heel,Rodrigo V. Portugal,Michael Schatz, (2016) Multivariate Statistical Analysis of Large Datasets: Single Particle Electron Microscopy. Open Journal of Statistics,06,701-739. doi: 10.4236/ojs.2016.64059

Jean-Pierre Bretaudière (1946-2008):

The Early Days of Multivariate Statistics in Electron Microscopy

Jean-Pierre Bretaudière (Figure A1) had a very fundamental influence in the early days of multivariate statistical analysis of the images of single particles in electron microscopy, an influence that was not well documented in terms of co-authorships or in other written documents. Let me explain how he was involved in the “intelligent averaging” of specific views of single particle images. Averaging of a great many “unit cells” of molecules arranged as a helical assembly [70] or as a two-dimensional crystal [71] , had at the time made quite an impression in the field and many electron microscopy groups worldwide thus started image processing projects. The first ideas of applying the concept of averaging to the images of individual macromolecules had been formulated [72] and implemented in Owen Saxton’s IMPROC software system. It was not yet clear at the time how best to find the 3D structure of a macromolecular complex based on individual two-dimensional projection images of the molecules. Would single-particle tomography such as proposed by the group of Hoppe [8] be the way towards, or did one first have to find all the different views presented by a macromolecular complex? Around 1980, we were still a decade away from answering such questions.

Since 1976, I had been employed as a project scientist in the electron microscopy group of Professor Ernst van Bruggen of the Biochemistry Department of the University of Groningen, The Netherlands. Erni van Bruggen had been most impressed by the early “averaging” successes of the group around Aaron Klug in Cambridge and wanted to incorporate those techniques in his research. It was my task to develop image processing software for data collected either with the transmission electron microscope (TEM) or the Scanning version of that instrument (a “STEM”). In the fall of 1979, I went to visit Albany, for a 6-week period, to work with Joachim Frank on automatically finding sub-populations of images in a heterogeneous population as discussed in the main paper. We had met two years earlier at an EMBO image processing course in Basel and we had been corresponding ever since. We both agreed that when averaging images of individual macromolecules, the presence of a mixture of different views was a serious and important complication: a challenging problem in need of a solution. I travelled to Albany taking with me two different sets of images in which different views could already be distinguished visually. One set consisted of images of worm hemoglobin (two different preparations: one with a clearly visible extra subunit in the middle and one without) and the other dataset of arthropod 4 × 6 hemocyanins where the difference between “flip” and “flop” could be directly seen (Figure A2).

What followed were many nights in the windowless catacombs of the New York State Department of Health (Division Laboratories and Research, now the Wadsworth Centre) beneath the Empire State Plaza. During these late sessions I met Jean-Pierre Bretaudière (“JP”) while working at the same PDP-11 45 computer (Figure A3 and Figure A4). We first talked about other issues than science, such as his personal ties to Paris (Figure A1)

where I had spent most of 1971 as a student. We also discussed our actual work and I explained the first attempts Joachim and I were making at separating subsets of images from the overall population of molecular images. Jean-Pierre explained what he was doing at the State Department of Health: quality control of medical laboratories in the state of New York. The Health Department would send out identical blood samples to many different medical laboratories. The results of the various chemical tests from the different labs were then compared using a French multivariate statistical technique called “l’analyse des correspondances” [73] . At the time I had never heard of multivariate statistical analysis (“MSA”), let alone of Benzécri’s “correspondence analysis” [74] .

Jean-Pierre was immediately convinced that his “AFC” program (for: “Analyse Factorielle des Correspondances”) would be able to help solve our problem and handle our “massive” amount of image data. The AFC program was a modified version of the correspondence analysis program “CORAN” by Jean Paul Benzécri [74] .

What was “massive” in this context was a very relative and very dated concept: at the time we had less than 64 images of 64 × 64 pixels = 4096 pixels each; that is a total data set size of less than one megabyte (each image being 16 Kbyte in floating point format). An uncompressed picture from even the cheapest digital camera today is much larger than one megabyte! However, it was characteristic for those early days in computing, that the 16-bit PDP-11 computer would only address 2¹⁶ bytes = 64 Kbytes. It was thus difficult to fit any “major” computational problem in the central memory of such a computer! The critical memory requirement in our case was the “core” needed for the square variance-covariance matrix (see main article) which, for 64 images would already occupy 16 Kbyte of the precious available address space.

As we sat together with Joachim the following morning to discuss the matter, we were all convinced that this was very much worth trying. Joachim wrote a small conversion program to allow JP’s AFC program to read the aligned images produced by the SPIDER program. The result of this first correspondence analysis of single particles was an instant success!

The first trial was on a giant annelid hemoglobin data set extracted from images of hemoglobins from two different species. The giant annelid hemoglobin of Oenone fulgida visibly had extra density in the centre [75] compared to the well known giant hemoglobin of the common earthworm Lumbricus terrestris which rather had a “hole” in the centre of its double-ring geometry. The separation of the two species, artificially mixed for this analysis, was so obvious that we immediately moved to a single data set with real internal variations. We thus moved to the second data set I had brought from Groningen, negative-stain images of Limulus polyphemus 4x6-meric hemocyanins (a dissociation product of the full 8x6-meric native hemocyanin).

On the correspondence analysis maps of these images four clear groups (“classes”) were visible. These reflected the “flip and flop” versions of the 4 × 6 oligomer where the right hand 2 × 6 part can be shifted up or down with respect to the left-hand side 2 × 6-mer. This behaviour had already been anticipated based on visual inspection of the images (Figure A2). However, that effect would have led to a subdivision into two groups rather than four. The other effect, which I had not anticipated, split both the “flip” and the “flop” groups in two, namely that of “rocking”. The four hexamers constituting the 4 × 6-mer were apparently not in one plane which led to two stable “rocking” positions where 3 out of 4 hexamers would be in direct contact with the carbon support film. This explanation, however, was not so straightforward because the alignment algorithm used at that time would itself introduce a 180˚ ambiguity in the in-plane rotational alignment. Both the shape of the molecule and the flaws of the alignment scheme thus led to the same effect. This issue was eventually resolved after confusing argumentations with referees [11] [12] . The books that JP recommended [74] ; and especially [14] became my “bibles” for years to come (English versions of these books have since appeared making that literature more accessible).

In these early MSA results, we would print out two-dimensional maps of the positions of the images on the first factorial co-ordinates and then draw a circle around a cluster of images, call that a “class” and then sum the members of that class for further interpretation. This visual classification, after eigenvector-eigenvalue data reduction, obviously had its shortcomings and was impossible to perform when more than ~3 factorial axes were to be taken into account. It was JP who also pointed me in the direction of automatic classification and the wealth of literature on the subject [76] , which then led to the introduction of automatic Hierarchical Ascendant Classification, “HCA”, in electron microscopy [16] .

Jean-Pierre was a wild guy-bursting with energy, full of ideas and projects, always burning the candle at both ends. I have vivid memories of, after a late evening session at work and an even longer night in a bar, stumbling out of the bar roaring with laughter along with JP and Vicky, Joachim’s programmer who later became JP’s wife. JP was always exuberant in his love for life and all its adventures. Jean-Pierre, having seen the success of correspondence analysis in single-particle electron microscopy, became directly interested in EM image processing. He changed fields and moved to the University of Texas Medical Center in Houston, Texas, where he developed the SUPRIM image processing system for electron microscopy [77] . In Houston, being JP, he was proud to declare himself more Texan than the Texans: he bought a ten-gallon hat, a huge Lincoln with “BRETO” vanity plates, and a large gun to defend his home and family. I still vividly remember the self-satisfied smile on his face when he told me this...

The last time we spoke, already many years ago, was about the time he left Houston to return to France. We spoke of his health issues and about his plans to leave the single-particle EM field-again. JP was never afraid of rigorous decisions. To this day, I am sorry we did not include JP as a co-author of the first papers that resulted [11] [12] He was offered co-authorship but refused. Maybe we simply did not push him hard enough. His co-authorship would have reflected his essential early contributions to the ideas in this field. The only printed evidence of our early collaboration which remains is the abstract for the European electron microscopy meeting in Hamburg in 1982 mentioned above [76] .

The beautiful portrait of JP by Jan Galligan (Figure A1), which so clearly documents his joie de vivre, resurfaced in my archives a few years ago. After scanning it, I searched the web to find JP’s current email address hoping to put a smile on his face. Unfortunately, I failed to find a current address. Only recently, after receiving the sad message of Jean-Pierre’s death, did I learn he had moved to Madagascar, after selling his very successful French computer business-Brett Computers-moving away from western society’s rat-race. Some say that scientists are dry and boring people… There is no doubt that with JP no longer around, the world of science has become a slightly more boring small universe.

I am grateful to Jan Galligan and Robert Rej for letting me to use their pictures, and for their support and constructive suggestions (Marin van Heel; 2009).

Submit or recommend next manuscript to SCIRP and we will provide best service for you:

Accepting pre-submission inquiries through Email, Facebook, LinkedIn, Twitter, etc.

A wide selection of journals (inclusive of 9 subjects, more than 200 journals)

Providing 24-hour high-quality service

User-friendly online submission system

Fair and swift peer-review system

Efficient typesetting and proofreading procedure

Display of the result of downloads and visits, as well as the number of cited articles

Maximum dissemination of your research work

Submit your manuscript at: http://papersubmission.scirp.org/