Since the 70's researchers in statistics, psychology and biology, have developed methods to compare clusterings. If distance matrices between individual entities are available for both clusterings, it may be possible to directly correlate the pairwise distances 1. But more frequently, researchers are interested in knowing if the resulting groups are similar or not. It is also possible to have highly correlated distance matrices that give rise to very different partitions due to scale heterogeneity in the distance values. Hence, the methods presented in the literature have been focused in the comparison of partitions (also designated flat clusterings), neglecting the closeness relationships between clusters. There are two main families of methods comparing partitions. One evaluating pairwise agreement (Rand, Adjusted Rand, Fowlkes-Mallows, Jaccard and Wallace indices) 23456, the other searching for clusterwise agreement (Larsen, Meila's variation of information and Van Dongen indices) 789. In both families, some methods are asymmetric, that is, the agreement of clustering A with B is different of the agreement of B with A 67. This asymmetry can be helpful if the symmetric methods are being effected by the different discriminatory power of the two clusterings. Clusterwise methods are computed from a contingency table (CT, Table 1) that contains the dual classification of each individual entity in both clusterings, while pairwise methods are computed from a 2 by 2 mismatch matrix (MM, Table 2), derivable from the CT. Each of the four cells of MM count the pairs of entities that belong or not to the same cluster in either of the two clusterings. None of the two matrices CT or MM contain any information about the relatedness of the different clusters.

Although the research in this area has produced many different methods, the classical methods are the most frequently referred, as an example, in a recent reference book on microarray data analysis, the only presented method to compare clusterings is equivalent to the Rand index 10. On the other hand there is no general consensus on the choice of the method to compare clusterings, and active research on alternative methods was motivated by microarray and systems biology approaches 11. It should be noted that the methods discussed here are not evaluating the quality or validating clustering algorithms. Instead the aim is to confront information of clusterings obtained from different data sources. It is also not a direct aim to achieve a combined better clustering closer to a hypothetical true classification. Nonetheless, these are possible secondary applications that are not tested in the present report. Additionally, researchers comparing clustering results should be aware that the measured levels of agreement could be strongly influenced by the inherent quality of the individual clusterings and by the type and quality of the datasets that originated the analysed clusterings.

The motivation to develop a new method stemmed from the observation that, for the available measures, when pairs of entities are in the same cluster on one clustering, and in different clusters on the other, it is considered irrelevant if these clusters are close neighbours or, on the contrary, very distant. A solution for such a problem was developed in a related subject, the quantification of the agreement of different observers performing a diagnostic test 12. When the test has multiple possible categories with an ordinal relation (of disease severity, for example), weights are attributed to different degrees of disagreement. Minimal (when one observer chooses one category close to the one chosen by the other observer), intermediate and maximal disagreement (when the two observers choose categories in the two extremes of the ordinal scale), and these contribute proportionally to the overall measure of agreement computed.

The use of a similar weighting strategy in the comparison of clusterings is not directly applicable. First, and in contrast to the observer agreement case, the two clusterings are not forced to have the same number of groups. Second, there is no predetermined correspondence between the clusters in both clusterings. Third, the closeness relationships between clusters are frequently more complex (needing two or more dimensions to be correctly represented) than the simple ordinal scale of diagnostic categories (an unidimensional representation). The main achievements of the proposed measure are the solutions for these three problems. It consists on the definition of a new way to record pairwise agreements in a Ranked Mismatch Matrix (RMM), enabling the combined accounting of partition and intercluster distance information in the computation of an overall clustering agreement measure. The new measure was named Ranked Adjusted Rand (RAR), because it can be considered an expansion of the previous Hubert and Arabie (HA) adjusted Rand index and both measures are equivalent when there is no intercluster distance information available for both clusterings.

The Methods section describes how to compute RAR from a Ranked Mismatch Matrix (RMM, represented in Table 3) and the quantities Mean Diagonal Deviation (MDD) and expected MDD under independence of clusterings (MDD^ind). MDD can be interpreted as the expected change in intercluster distance rank for a randomly chosen pair of entities. Considering one entity pair (a, b). If in clustering C, b belongs to the r^th closest cluster to a's cluster, then it is expected that in C', b is in the r ± (MDD^ind × K')^th closest cluster to a's cluster (K' is the number of clusters in C'). This kind of interpretation can be very useful if the aim is to predict the clusters obtained with one technique or data source using the clustering information obtained by a different technique or data source. MDD can take the value 1 only in a single situation: when in one clustering all entities are in the same clustering and in the other, every entity is in its own cluster, and all clusters are equally distant from each other. On the other hand, a MDD value of 0 corresponds to two clusterings with exactly identical partitions and equally ranked relative distances between clusters. The RAR values compare the observed MDD values with the theoretical MDD value if the assignment of entities to clusters was independent in both clusterings (the agreement in both clusterings would only be due to chance alone). The maximum value taken by RAR is 1, when MDD is 0. If MDD <MDD^ind, the average entity pair tends to have smaller intercluster distance rank changes from one clustering to the other than it would have in the independence situation. In this case RAR takes positive values, meaning that the clusterings are more similar than expected by chance agreement. If MDD > MDD^ind, RAR takes negative values, meaning that the deviation from perfect agreement is greater than expected by chance. The two last situations imply a very similar interpretation to the HA adjusted Rand case. RAR values are certainly less intuitive than interpreting simple Rand, but RAR provides more rich information about clustering agreement. RAR will be especially useful to distinguish situations in which HA or other measures are almost or completely identical. For these reasons we are proposing to use RAR in addition to previously available measures.

When both clusterings being compared are flat, that is, when there is no intercluster distance information for any of them, two entities can only be either in the same cluster or in equally dissimilar clusters. RMM becomes identical to MM. In that situation, 1-MDD is equal to the Rand index. Analogously, RAR becomes HA, since the correction for chance agreement is similar for both measures in the absence of any intercluster distance information. Proof of this equivalence is presented in Additional file 1.

A major potential application of RAR is the comparison of a flat clustering with other for which interclusters distance information is available. One can use previously developed partition comparison measures to do this but the distance information available is neglected. However, RAR is able to compare both clusterings including the partial distance information available. The resulting RMM will have 2 × (q+1) (or (p+1) × 2) dimensions and it is possible to evaluate if the mismatches of the flat clustering tend to originate mismatches with larger rank differences in the other clustering than the flat matches.

This and the previous sections discussed the use of RAR when there is a partial or total absence of distance information. However, in most of the situations that Rand or Adjusted Rand indexes have been used, information about distance was indeed available. For an example see reference 11. This is the most frequent situation when the partitions compared where produced by clustering algorithms. The clustering algorithm needs an inter-entity distance matrix, and this matrix is sufficient to derive the intercluster distances used for RAR computation. When the partitions are defined by classification methods, it may still be possible to have distance information, depending on the properties used to classify entities.

A positive feature of the R(i, j) is that it is unnecessary to define rules to deal with rank ties. If a cluster has two neighbour clusters at the same distance, they will have the same intercluster distance rank. The only consequence is that the maximum intercluster distance rank (p+1 or q+1) decreases with the number of ties. RMM will have p+1 = K rows and q+1 = K' columns in the absence of ties in intercluster distance ranks. Each tie in C will reduce one row and each tie in C' will reduce one column to RMM. A higher number of ties can be due to a more discrete intercluster distance function and will produce a reduced RMM that is more similar to MM. The existence of ties is then responsible for the approximation of RAR to HA. This is consistent with the fact that more ties are a consequence of a lower resolution of the metric used to define the intercluster distance function. The minimal resolution corresponds to a binary distance function that can be 0 (same cluster) or 1 (different cluster) – that is, when RAR is equal to HA, as discussed previously.

To clearly show the desirable properties of the RAR measure compared with previously available methods, four theoretical simple clusterings were created (Figure 1). One of the four, clustering A, is the original one, with 9 entities divided in 3 clusters. The position of the points in each clustering is relevant. Two points that are more distant are less similar. Clusterings B to D were originated from A by splitting the {1, 2, 3, 4} cluster in two. One of the resulting clusters kept the same location, while the other varies in size and location in the different clusterings. The coordinates and cluster identity of every corresponding nine entities in the four clusterings were used to compute RAR and other ten measures of clustering agreement between A and each of its transformed clusterings. The description and formulas of these additional measures are available in the corresponding references given in Table 4, together with the values computed for these examples.

The first point that these theoretical examples demonstrate is that RAR, contrary to the previous partition comparison measures, is able to detect a greater disagreement between clusterings if the entities causing the disagreement, besides changing the composition of the clusters change also the proximity relationships between clusters. That is shown by the difference in RAR value for the comparisons of A with B and A with C (Figure 1). All the other ten comparison measures consider B and C equally similar to A. In fact, the change in cluster composition from A to B is identical to the change from A to C. The difference is that the newly formed clusters in B are the closest neighbours, while in C they are the most distant clusters. As the discussed clusterings involve a small number of entities and clusters, and considering that the distance between points is proportional to the dissimilarity that was used to generate the clusterings, observation of Figure 1 clearly indicates that clustering B is more similar to A than C is.

On the contrary, RAR indicates that clustering D is more similar to A than B is. This arises because in D only one entity has a different location comparing with A. From A to B three entities changed position, although to the same relative location of the one entity cluster in D. Again, only RAR detected this difference, while all the other measures remained unchanged. This happens because RAR uses the intercluster distance ranks. The RMM comparing A with B will have more point pairs out of the diagonal than the comparison of A with D. In the first comparison three points changed their relative position, affecting the RMM position of 3 × 6 pairs of points. In the second comparison, only one point moved, hence, only 1 × 8 pairs of points can have different intercluster distance ranks than they would in a perfect match comparison. Consequently, RAR attributes more weight to the change from A to B than from A to D. From the point of view of partition comparison measures, B and D have equal differences relatively to A. They both result from A by splitting a cluster of 4 entities into one with 3 and other with 1 entity alone. This is the reason why the 10 partition comparison measures in Table 4 are not able to distinguish between the similarity of A with B and of A with D.

The two small comparisons of the previous section are extreme cases where the advantages of RAR were demonstrated, since it was able to detect differences between clusterings that none of the previously available methods where able to detect. But in realistic data sets, it is expected that a variable number of entities change their cluster membership and their relative position simultaneously. Additionally, some entity changes may contribute to make clusterings more similar while others may differentiate them. For these reasons, larger clusterings where simulated, also with more extensive entity shuffling. A complete description of these simulations is provided in Additional file 2. Briefly, five factors were systematically varied in simulated clustering comparisons: 1) number of entities, 2) number of clusters, 3) cluster size distribution, 4) fraction of entities changing cluster membership and relative position and 5) extension of change in relative position. The only factor that had an effect on the final RAR values was the fraction of entities changing cluster membership and location, producing a linear correlation coefficient of r = -0.918. The number of entities (r = 0.077), number of clusters (r = -0.109), and the cluster size distribution (r = -0.032) had negligible impact on RAR values. These low correlations support the conclusion that RAR values are not systematically influenced neither by the number of entities and clusters being compared nor by the distribution of cluster sizes. As the entities changing cluster membership were randomly selected from every possible cluster, the change in relative position of some entities could be balanced by entities moving in the opposite direction. Consequently, varying the extension of change in relative position produced highly variable results and a low correlation with RAR values (r = -0.016). To evaluate more precisely the influence of this factor on RAR values, a partial correlation analysis was performed on the relation between RAR values, HA values (that can be interpreted as RAR values without intercluster distance information) and the net change in entity relative position (measured by the correlation coefficient between the distance matrices of the two clusterings being compared). The results, presented in detail on supplementary material, show that RAR integrates independent information contained in the HA Index and in the correlation coefficient between distance matrices. The partial correlation of RAR with both factors are strong and positive (0.758 and 0.720), which means that both a higher fraction of entities changing cluster membership and a higher net change in entity relative position independently induce higher RAR values.

To substantiate the general applicability of RAR, three examples with biological data are presented that compare 1) two microbial typing methods, 2) two gene regulatory network distances and 3) microarray gene expression data with pathway information. The first example is presented in the main text while the other two are included in Additional file 3.

Typing methods are major tools for the epidemiological characterization of bacterial pathogens, allowing the determination of clonal relationships between isolates based on their genotypic or phenotypic characteristics. Since typing schemes analyze different phenotypic or genotypic properties of bacteria, if some congruence between the methods is found, it suggests that a phylogenetic signal is being recovered by both methods, allowing greater confidence about evolutionary hypothesis or clonal dispersion of the strains under study. The same collection of bacterial isolates can be typed by different methodologies and it becomes of great epidemiological and evolutionary importance to understand the relationships between the clusters of isolates defined by the different methods. To this end we have recently evaluated the usefulness of a set of measures to quantitatively describe these relationships 13.

We analyzed the data generated by the characterization of a collection of 325 macrolide-resistant Streptococcus pyogenes 1415. This collection was characterized by emm sequencing, that generates groups of isolates differing by less than 92% in their DNA sequence and by comparison of the patterns generated after digestion of total DNA with the SmaI endonuclease and separation by pulsed-field gel electrophoresis (PFGE). Dice coefficient was used to compute dissimilarity between PFGE band patterns, enabling the subsequent hierarchical clustering with average linkage. Measurement of the agreement between the emm classification and PFGE clusterings for the same data set has already been done using HA and Wallace indices (W) 13. Wallace index of clustering A relatively to B is the probability that two entities are in the same cluster in B, knowing they were in the same cluster in A. It is a pairwise asymmetric clustering agreement measure 6. For the present work, PFGE clusterings were produced for 70 different Dice dissimilarity thresholds, covering all the possible values. Each of these clusterings was compared with the emm classification through RAR, HA and Wallace indices.

The dendrogram built with PFGE and emm type classification data is shown in Figure 1 of the Additional file 3. Although major agreements for half of the emm types (1,4, 9, 11, 12 e 22) are identifiable within the dendrogram, it becomes a hard task to quantify the overall concordance including the other types. In order to compare the emm classification and the clustering with PFGE it is first required to define the threshold that maximizes the agreement for the two microbial typing methods. Figure 2 shows the values of RAR, HA and Wallace indices for the range of Dice dissimilarity thresholds used on the PFGE clustering.

A clustering C is a partition of the set of objects D, with n elements (identified below by the letters i and j), into sets (clusters) C₁, C₂,...C_K, with n₁, n₂,...n_K number of entities, all greater than 0. The task of measuring clustering agreement arises when, for the same set D, two different methods are used to produce two different clusterings, C and C', with K and K' clusters each. To evaluate the overlap of the two partitions, a contingency table is built, where every element of D contributes to the cell of the corresponding clusters in both C and C' as shown in Table 1. Focusing on the pairwise agreement, the information in CT can be further condensed in a mismatch matrix represented in Table 2, where a, b, c and d represent the counts of entity pairs that fall in each of the four possible categories. For example, entity pairs in the b category are in the same cluster in C but in different clusters in C'. The sum of a, b, c and d is n(n-1)/2, the total number of unique entity pairs.

Hubert and Arabie proposed an adjusted Rand index to quantify clustering agreement 4:

H A ( C , C ′ ) = a + d − n c a + b + c + d − n c       ( 1 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGibascqWGbbqqcqGGOaakcqWGdbWqcqGGSaalcuWGdbWqgaqbaiabcMcaPiabg2da9maalaaabaGaemyyaeMaey4kaSIaemizaqMaeyOeI0IaemOBa42aaSbaaSqaaiabdogaJbqabaaakeaacqWGHbqycqGHRaWkcqWGIbGycqGHRaWkcqWGJbWycqGHRaWkcqWGKbazcqGHsislcqWGUbGBdaWgaaWcbaGaem4yamgabeaaaaGccaWLjaGaaCzcamaabmaabaGaeGymaedacaGLOaGaayzkaaaaaa@4B69@

Where n_c is the correction for chance agreement, corresponding to the expected sum of a and d if C and C' where totally independent clusterings:

n c = n ( n 2 + 1 ) − ( n + 1 ) ∑ k = 1 K n k 2 − ( n + 1 ) ∑ k ′ = 1 K ′ n ′ k ′ 2 + 2 ∑ k = 1 K ∑ k ′ = 1 K ′ n k 2 n ′ k ′ 2 n 2 ( n − 1 )       ( 2 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGUbGBdaWgaaWcbaGaem4yamgabeaakiabg2da9maalaaabaGaemOBa4MaeiikaGIaemOBa42aaWbaaSqabeaacqaIYaGmaaGccqGHRaWkcqaIXaqmcqGGPaqkcqGHsislcqGGOaakcqWGUbGBcqGHRaWkcqaIXaqmcqGGPaqkdaaeWbqaaiabd6gaUnaaDaaaleaacqWGRbWAaeaacqaIYaGmaaGccqGHsislcqGGOaakcqWGUbGBcqGHRaWkcqaIXaqmcqGGPaqkdaaeWbqaaiqbd6gaUzaafaWaa0baaSqaaiqbdUgaRzaafaaabaGaeGOmaidaaOGaey4kaSIaeGOmaiZaaabCaeaadaaeWbqaamaalaaabaGaemOBa42aa0baaSqaaiabdUgaRbqaaiabikdaYaaakiqbd6gaUzaafaWaa0baaSqaaiqbdUgaRzaafaaabaGaeGOmaidaaaGcbaGaemOBa4gaaaWcbaGafm4AaSMbauaacqGH9aqpcqaIXaqmaeaacuWGlbWsgaqbaaqdcqGHris5aaWcbaGaem4AaSMaeyypa0JaeGymaedabaGaem4saSeaniabggHiLdaaleaacuWGRbWAgaqbaiabg2da9iabigdaXaqaaiqbdUealzaafaaaniabggHiLdaaleaacqWGRbWAcqGH9aqpcqaIXaqmaeaacqWGlbWsa0GaeyyeIuoaaOqaaiabikdaYiabcIcaOiabd6gaUjabgkHiTiabigdaXiabcMcaPaaacaWLjaGaaCzcamaabmaabaGaeGOmaidacaGLOaGaayzkaaaaaa@7BB3@

Milligan and Cooper 20 and more recently Steinley 21 performed comparative studies of several pairwise clustering agreement criteria. They found HA the criterion with the most desirable properties, especially the zero expected value in the case of independent clusterings and the robustness to changes in cluster number and cluster size heterogeneity. The basic principle of HA is to compute the fraction of entity pairs in the diagonal of MM, because those pairs are the ones contributing to the general agreement. The pairs in b and c have a null contribution to the agreement. This fraction must be corrected for the expected chance agreement.

To include the intercluster distance information, the entity pairs in a should continue to have a maximum contribution to the overall agreement, but b, c and d entity pairs should have different contributions according to the degree of mismatch in each of the two clusterings. First an intercluster distance rank function R is defined for every pair of entities (i, j) of a data set D (expression 3).

R(i, j) = (x, y): i, j ∈ {1,2,...n}; x ∈ {1,2,...K - 1}; y ∈ {1,2,...K' - 1}     (3)

R(i, j) = (x, y), means that in clustering C, entity j is in the x^th cluster closer to the one of entity i, and in clustering C', the cluster of entity j is the y^th closer to the cluster of i. In the case i and j are in the same cluster in C, x will be 0. If i and j are in the same cluster in C', y will be 0. The distance between two clusters is here measured as the average distance between their entities. This is only possible when distances between every pair of entities are available. According to the problem, other intercluster distance function can be defined. For instance the standard single, complete or other linkage functions of hierarchical clustering can be used. In the absence of any distance information, the distance between a cluster and itself is 0 and between two different clusters is 1. Additionally, the intercluster distance definition does not have to be the same in the two clusterings being compared. These definitions allow the RAR method to be applied to any pair of clusterings. With the help of the intercluster distance rank function the Ranked Mismatch Matrix (RMM), represented in Table 3, can be computed, with the general element rmm_x,y defined as:

r m m x , y = ∑ i = 1 n ∑ j = 1 n H ( i ≠ j ) H ( R ( i , j ) = ( x − 1 , y − 1 ) )       ( 4 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGYbGCcqWGTbqBcqWGTbqBdaWgaaWcbaGaemiEaGNaeiilaWIaemyEaKhabeaakiabg2da9maaqahabaWaaabCaeaacqWGibascqGGOaakcqWGPbqAcqGHGjsUcqWGQbGAcqGGPaqkcqWGibascqGGOaakcqWGsbGucqGGOaakcqWGPbqAcqGGSaalcqWGQbGAcqGGPaqkcqGH9aqpcqGGOaakcqWG4baEcqGHsislcqaIXaqmcqGGSaalcqWG5bqEcqGHsislcqaIXaqmcqGGPaqkcqGGPaqkaSqaaiabdQgaQjabg2da9iabigdaXaqaaiabd6gaUbqdcqGHris5aaWcbaGaemyAaKMaeyypa0JaeGymaedabaGaemOBa4ganiabggHiLdGccaWLjaGaaCzcamaabmaabaGaeGinaqdacaGLOaGaayzkaaaaaa@6280@

H(x) is a Heaviside step function that takes the value 1 when x is true and 0 otherwise. The double sum includes the equal entity pairs of type (i, i) and the repeated entity pairs of types (i, j) and (j, i). The pairs of the first type do not contribute to the final sum due to the Heaviside function H(i≠j). The repeated pairs (differing only by the order of the entities inside the pair) need to be accounted in the sum because, for each of the individual clusterings, the intercluster distance rank is not necessarily symmetric. As an example, cluster A may be the closest neighbour of cluster B, but the closest neighbour of cluster B may be C and not A. In RMM, intercluster distance rank information for every pair of entities is recorded without identifying which clusters are separated at what rank distance. This is important because for each cluster, the i^th neighbour cluster can be different. In this way, the intercluster distance information can be integrated with the partition comparison without the need of a strict ordinal relationship between clusters (like the example of disease severity referred in the introduction), of a known cluster correspondence between clusterings or of an equal number of clusters in both clusterings.

For two very similar clusterings, the majority of the entity pairs would contribute for elements close to the matrix diagonal. Even if RMM is not square, an alternative geometrical diagonal can be traced, linking the centre of the rmm_1,1 element (with coordinates (0,0)) with the center of the rmm_p+1,q+1 element (with coordinates (p, q)) If, on the contrary, the clusterings disagree to a large extent, most entity pairs will be far from the diagonal, concentrated around rmm_p+1,1 and rmm_1,q+1. From these considerations it immediately follows that a good measure of cluster disagreement is the Mean Diagonal Deviation (MDD) for all the entity pairs in RMM.

M D D = ∑ i = 1 p + 1 ∑ j = 1 q + 1 r m m i , j ⋅ | i − 1 p − j − 1 q | n 2 − n       ( 5 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGnbqtcqWGebarcqWGebarcqGH9aqpdaWcaaqaamaaqahabaWaaabCaeaacqWGYbGCcqWGTbqBcqWGTbqBdaWgaaWcbaGaemyAaKMaeiilaWIaemOAaOgabeaaaeaacqWGQbGAcqGH9aqpcqaIXaqmaeaacqWGXbqCcqGHRaWkcqaIXaqma0GaeyyeIuoakiabgwSixpaaemaabaWaaSaaaeaacqWGPbqAcqGHsislcqaIXaqmaeaacqWGWbaCaaGaeyOeI0YaaSaaaeaacqWGQbGAcqGHsislcqaIXaqmaeaacqWGXbqCaaaacaGLhWUaayjcSdaaleaacqWGPbqAcqGH9aqpcqaIXaqmaeaacqWGWbaCcqGHRaWkcqaIXaqma0GaeyyeIuoaaOqaaiabd6gaUnaaCaaaleqabaGaeGOmaidaaOGaeyOeI0IaemOBa4gaaiaaxMaacaWLjaWaaeWaaeaacqaI1aqnaiaawIcacaGLPaaaaaa@630B@

The quantity inside the modulus is the normalized distance of the element (i, j) to the RMM diagonal, such that for the more distant elements (rmm_1,q+1 and rmm_p+1,1) it takes the value of 1. Consequently, the maximum value of MDD is also 1. The modulus implies that MDD is always greater or equal to 0. To obtain a measure of agreement between clusterings it is enough to compute 1-MDD, although this quantity is not yet corrected for chance agreement. To perform this correction, the expected MDD value under independence of clusterings C and C' (conditional on the marginals of CT and on the intercluster ranked distances) must be known. To compute this MDD^ind it is first necessary to build RMM^ind according to:

r m m x , y i n d = ( ∑ i = 1 K ∑ j = 1 K ( H ( R ( ( ∀ s : s ∈ C i ) , ( ∀ t : t ∈ C j ) ) = ( x − 1 , ⋅ ) ) ⋅ n i ⋅ n j − H ( i = j ) ⋅ n i ) ) × ( ∑ i = 1 K ′ ∑ j = 1 K ′ ( H ( R ( ( ∀ s : s ∈ C i ′ ) , ( ∀ t : t ∈ C j ′ ) ) = ( ⋅ , y − 1 ) ) ⋅ n i ′ ⋅ n j ′ − H ( i = j ) ⋅ n i ′ ) ) / ( n 2 − n )       ( 6 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqabeGabaaabaGaemOCaiNaemyBa0MaemyBa02aa0baaSqaaiabdIha4jabcYcaSiabdMha5bqaaiabdMgaPjabd6gaUjabdsgaKbaakiabg2da9maabmaabaWaaabCaeaadaaeWbqaaiabcIcaOiabdIeaijabcIcaOiabdkfasjabcIcaOiabcIcaOiabgcGiIiabdohaZjabcQda6iabdohaZjabgIGiolabdoeadnaaBaaaleaacqWGPbqAaeqaaOGaeiykaKIaeiilaWIaeiikaGIaeyiaIiIaemiDaqNaeiOoaOJaemiDaqNaeyicI4Saem4qam0aaSbaaSqaaiabdQgaQbqabaGccqGGPaqkcqGGPaqkcqGH9aqpcqGGOaakcqWG4baEcqGHsislcqaIXaqmcqGGSaalcqGHflY1cqGGPaqkcqGGPaqkcqGHflY1cqWGUbGBdaWgaaWcbaGaemyAaKgabeaakiabgwSixlabd6gaUnaaBaaaleaacqWGQbGAaeqaaOGaeyOeI0IaemisaGKaeiikaGIaemyAaKMaeyypa0JaemOAaOMaeiykaKIaeyyXICTaemOBa42aaSbaaSqaaiabdMgaPbqabaaabaGaemOAaOMaeyypa0JaeGymaedabaGaem4saSeaniabggHiLdaaleaacqWGPbqAcqGH9aqpcqaIXaqmaeaacqWGlbWsa0GaeyyeIuoakiabcMcaPaGaayjkaiaawMcaaiabgEna0cqaamaalyaabaWaaeWaaeaadaaeWbqaamaaqahabaGaeiikaGIaemisaGKaeiikaGIaemOuaiLaeiikaGIaeiikaGIaeyiaIiIaem4CamNaeiOoaOJaem4CamNaeyicI4Saem4qam0aaSbaaSqaaiabdMgaPbqabaaccaGccqWFYaIOcqGGPaqkcqGGSaalcqGGOaakcqGHaiIicqWG0baDcqGG6aGocqWG0baDcqGHiiIZcqWGdbWqdaWgaaWcbaGaemOAaOgabeaakiab=jdiIkabcMcaPiabcMcaPiabg2da9iabcIcaOiabgwSixlabcYcaSiabdMha5jabgkHiTiabigdaXiabcMcaPiabcMcaPiabgwSixlabd6gaUnaaBaaaleaacqWGPbqAaeqaaOGae8NmGiQaeyyXICTaemOBa42aaSbaaSqaaiabdQgaQbqabaGccqWFYaIOcqGHsislcqWGibascqGGOaakcqWGPbqAcqGH9aqpcqWGQbGAcqGGPaqkcqGHflY1cqWGUbGBdaWgaaWcbaGaemyAaKgabeaakiab=jdiIkabcMcaPaWcbaGaemOAaOMaeyypa0JaeGymaedabaGafm4saSKbauaaa0GaeyyeIuoaaSqaaiabdMgaPjabg2da9iabigdaXaqaaiqbdUealzaafaaaniabggHiLdaakiaawIcacaGLPaaaaeaadaqadaqaaiabd6gaUnaaCaaaleqabaGaeGOmaidaaOGaeyOeI0IaemOBa4gacaGLOaGaayzkaaaaaaaacaWLjaGaaCzcamaabmaabaGaeGOnaydacaGLOaGaayzkaaaaaa@E72D@

MDD^ind is then computed like MDD (expression 5), changing RMM elements by those of RMM^ind. RAR is the correction of (1-MDD) for chance agreement and is the result of the following expression:

R A R = M D D i n d − M D D M D D i n d       ( 7 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGsbGucqWGbbqqcqWGsbGucqGH9aqpdaWcaaqaaiabd2eanjabdseaejabdseaenaaCaaaleqabaGaemyAaKMaemOBa4MaemizaqgaaOGaeyOeI0Iaemyta0KaemiraqKaemiraqeabaGaemyta0KaemiraqKaemiraq0aaWbaaSqabeaacqWGPbqAcqWGUbGBcqWGKbazaaaaaOGaaCzcaiaaxMaadaqadaqaaiabiEda3aGaayjkaiaawMcaaaaa@483C@

Functions to compute the RAR measure for any two clusterings were implemented in MATLAB (Release 14), and are available in Additional file 4 or at the toolbox's webpage 22.