Multispecies coalescent process

Multispecies Coalescent Process is a stochastic process model that describes the genealogical relationships for a sample of DNA sequences taken from several species. It represents the application of coalescent theory to the case of multiple species. The multispecies coalescent results in cases where the relationships among species for an individual gene (the gene tree) can differ from the broader history of the species (the species tree). It has important implications for the theory and practice of phylogenetics and for understanding genome evolution.

A gene tree is a binary graph that describes the evolutionary relationships between a sample of sequences for a non-recombining locus. A species tree describes the evolutionary relationships between a set of species, assuming tree-like evolution. However, several processes can lead to discordance between gene trees and species trees. The Multispecies Coalescent model provides a framework for inferring species phylogenies while accounting for ancestral polymorphism and gene tree-species tree conflict. The process is also called the Censored Coalescent.

Discordance between gene trees and the species tree can lead to the cases where characters that appear to be homoplastic (to be gained or lost independently in separate lineages) given the relationships among species when those characters actually has a single origin; this phenomenon is called hemiplasy. Many studies that have focused on hemiplasy have focused on genomic characters like nucleotide or amino acid substitutions, indels, or karyotypic differences (though the last of these is thought to be less subject to hemiplasy than many other characters). However, phenotypic characters also have the potential to exhibit hemiplasy.

Gene tree-species tree congruence

Illustration of the multispecies coalescent showing the relationship between the species tree (black outline) and gene trees (dashed red lines embedded in the species tree). The time between the two speciation events (T, measured in coalescent units) can be used to calculate the probability of the four possible gene trees (using the equations shown). Note that two of the gene trees are topologically identical but they differ in the times at which lineages coalesce.

If we consider a rooted three-taxon tree, the simplest non-trivial phylogenetic tree, there are three different tree topologies but four possible gene trees. The existence of four distinct gene trees despite the smaller number of topologies reflects the fact that there are topologically identical gene tree that differ in their coalescent times. In the type 1 tree the alleles in species A and B coalesce after the speciation event that separated the A-B lineage from the C lineage. In the type 2 tree the alleles in species A and B coalesce before the speciation event that separated the A-B lineage from the C lineage (in other words, the type 2 tree is a deep coalescence tree). The type 1 and type 2 gene trees are both congruent with the species tree. The other two gene trees differ from the species tree; the two discordant gene trees are also deep coalescence trees.

The distribution of times to coalescence is actually continuous for all of these trees. In other words, the exact coalescent time for any two loci with the same gene tree may differ. However, it is convenient to break up the trees based on whether the coalescence occurred before or after the earliest speciation event.

Given the internal branch length in coalescent units it is straightforward to calculate the probability of each gene tree. For diploid organisms the branch length in coalescent units is the number of generations between the speciation events divided by twice the effective population size. Since all three of the deep coalescence tree are equiprobable and two of those deep coalescence tree are discordant it is easy to see that the probability that a rooted three-taxon gene tree will be congruent with the species tree is:

${\begin{aligned}P(congruence)&=1-{\frac {2}{3}}\exp(-T)=1-{\frac {2}{3}}\exp(-{\frac {t}{2N_{e}}})\end{aligned}}$

Where the branch length in coalescent units (T) is also written in an alternative form: the number of generations (t) divided by twice the effective population size (N_e). Pamilo and Nei also derived the probability of congruence for rooted trees of four and five taxa as well as a general upper bound on the probability of congruence for larger trees. Rosenberg followed up with equations used for the complete set of topologies (although the large number of distinct phylogenetic trees that becomes possible as the number of taxa increases makes these equations impractical unless the number of taxa is very limited).

Examples of the differences between hemiplasy (which requires gene tree-species tree differences) and true homoplasy (which can occur on a gene tree that is congruent with the species tree or a gene tree that is the discordant with tree species tree). This example shows the origins of some trait (blue). The presence (+) or absence (-) of the trait in each species is indicated at the top of the figure. Note that homoplasy can also reflect a single origin followed by a loss.

The phenomenon of hemiplasy is a natural extension of the basic idea underlying gene tree-species tree discordance. If we consider the distribution of some character that disagrees with the species tree it might reflect homoplasy (multiple independent origins of the character or a single origin followed by multiple losses) or it could reflect hemiplasy (a single origin of the trait that is associated with a gene tree that disagrees with the species tree).

The phenomenon called incomplete lineage sorting (often abbreviated ILS in the scientific literatures) is linked to the phenomenon. If we examine the illustration of hemiplasy with using a rooted four-taxon tree (see image to the right) the lineage between the common ancestor of taxa A, B, and C and the common ancestor of taxa A and B must be polymorphic for the allele with the derived trait (e.g., a transposable element insertion) and the allele with the ancestral trait. The concept of incomplete lineage sorting ultimately reflects on persistence of polymorphisms across one or more speciation events.

Mathematical description of the multispecies coalescent

The probability density of the gene trees under the multispecies coalescent model is discussed along with its use for parameter estimation using multi-locus sequence data.

Assumptions

The species phylogeny is assumed to be known. Complete isolation after species divergence, with no migration, hybridization, or introgression is also assumed. We assume no recombination so that all the sites within the locus share the same gene tree (topology and coalescent times).

Data and model parameters

The model and implementation of this method can be applied to any species tree. As an example, the species tree of the great apes: human (H), chimpanzee (C), gorilla (G) and orangutan (O) is considered. The topology of the species tree, (((HC)G)O)), is assumed known and fixed in the analysis (Figure 1). Let $D=\{D_{i}\}$ be the entire data set, where ${D_{i}}$ represent the sequence alignment at locus $i$, with $i=1,2,\ldots ,L$ for a total of $L$ loci.

The population size of a current species is considered only if more than one individual is sampled from that species at some loci.

The parameters in the model for the example of Figure 1 include the three divergence times $\tau _{HC}$, $\tau _{HCG}$ and $\tau _{HCGO}$ and population size parameters $\theta _{H}$ for humans; $\theta _{C}$ for chimpanzees; and $\theta _{HC}$, $\theta _{HCG}$ and $\theta _{HCGO}$ for the three ancestral species.

The divergence times ($\tau$'s) are measured by the expected number of mutations per site from the ancestral node in the species tree to the present time (Figure 1 of Rannala and Yang, 2003).

Therefore, the parameters are $\Theta =\{\theta _{H},\theta _{C},\theta _{HC},\theta _{HCG},\theta _{HCGO},\tau _{HC},\tau _{HCG},\tau _{HCGO}\}$.

Likelihood-based inference

The gene genealogy $G_{i}$ at each locus $i$ is represented by the tree topology $T_{i}$ and the coalescent times $t_{i}$. Given parameters $\Theta$, the probability distribution of $G_{i}=\{T_{i},t_{i}\}$ is specified by the coalescent process under the model given by

$f(G\mid \Theta )=\prod _{i}f(G_{i}\mid \Theta )=\prod _{i}f(T_{i},t_{i}\mid \Theta )$

The probability of data $D_{i}$ given the gene tree and coalescent times (and thus branch lengths) at the locus, $f(D_{i}\mid G_{i})$ is the Felsenstein's phylogenetic likelihood. Due to the assumption of independent evolution across the loci,

$f(D\mid G)=\prod _{i}f(D_{i}\mid G_{i})$

By Bayesian inference based on the joint conditional distribution

$f(\Theta ,G\mid D)\propto f(D\mid G)f(G\mid \Theta )f(\Theta ).$

Then, the posterior distribution of $\Theta$ is given by

$f(\Theta \mid D)=\int f(\Theta ,G\mid D)dG,$

where the integration represents summation over all possible gene tree topologies and integration over the coalescent times at each locus.

Distribution of gene genealogies

The joint distribution of $f(T_{i},t_{i}\mid \Theta )$ is derived directly in this section. Two sequences from different species can coalesce only in one populations that are ancestral to the two species. For example, sequences H and G can coalesce in populations HCG or HCGO, but not in populations H or HC. The coalescent processes in different populations are different.

For each population, the genealogy is traced backward in time, until the end of the population at time $\tau$, and the number of lineages $(m)$ entering the population and the number of lineages leaving it $(n)$ are recorded. For example, $m=3,n=2,$ and $\tau =\tau _{HC}$, for population H (Table 1). This process is called a censored coalescent process because the coalescent process for one population may be terminated before all lineages that entered the population have coalesced. If $n\geq 1$ the population consists of $n$ disconnected subtrees or lineages.

With one time unit defined as the time taken to accumulate one mutation per site, any two lineages coalesce at the rate ${\frac {2}{\theta }}$. The waiting time $t_{j}$ until the next coalescent event, which reduces the number of lineages from $j$ to $j-1$ has exponential density

$f(t_{j})={\frac {j(j-1)}{2}}{\frac {2}{\theta }}\exp\{-{\frac {j(j-1)}{2}}{\frac {2}{\theta }}t_{j}\},\quad j=m,m-1,\ldots ,n+1$

If $n\geq 1$, the probability that no coalescent event occurs between the last one and the end of the population at time $\tau$; i.e. during the time interval $\tau -(t_{m}+t_{m-1}+\ldots +t_{n+1})$. This probability is $\exp\{-{\frac {n(n-1)}{\theta }}[\tau -(t_{m}+t_{m-1}+\ldots +t_{n+1})]$ and is 1 if $n=1$.

(Note: One should recall that the probability of no events over time interval $t$ for a Poisson process with rate $\lambda$ is $e^{-\lambda t}$. Here the coalescent rate when there are $n$ lineages is $\lambda ={\frac {n(n-1)}{\theta }}$.)

In addition, to derive the probability of a particular gene tree topology in the population, if a coalescent event occurs in a sample of $j$ lineages, the probability that a particular pair of lineages coalesce is $1/{\binom {j}{2}}=2/j(j-1),\quad j=m,m-1,\ldots ,n+1$.

Multiplying these probabilities together, the joint probability distribution of the gene tree topology in the population and its coalescent times $t_{m},t_{m+1},\ldots ,t_{n+1}$ as

$\prod _{j=n+1}^{m}{\Big [}{\frac {2}{\theta }}\exp {\Big \{}-{\frac {j(j-1)}{\theta }}t_{j}{\Big \}}{\Big ]}\exp {\Big \{}-{\frac {n(n-1)}{\theta }}(\tau -(t_{m}+t_{m+1}+\ldots +t_{n+1})){\Big \}}$.

The probability of the gene tree and coalescent times for the locus is the product of such probabilities across all the populations. Therefore, the gene genealogy of Figure 1, we have

${\begin{aligned}f(G_{i}\mid \Theta )&=[2/\theta _{H}\exp\{-6t_{3}^{(H)}/\theta _{H}\}\exp\{-2(\tau _{HC}-t_{3}^{(H)})/\theta _{H}\}]\\&{}\times [2/\theta _{C}\exp\{-2t_{2}^{(C)}/\theta _{C}\}]\\&{}\times [2/\theta _{HC}\exp\{-6t_{3}^{HC}/\theta _{HC}\}]\times [2/\theta _{HC}\exp\{-2t_{2}^{HC}/\theta _{HC}\}]\\&{}\times [\exp\{-2(\tau _{HCG}-\tau _{HG}-(t_{3}^{HC}+t_{2}^{HC}))/\theta _{HCG}\}]\\&{}\times [2/\theta _{HCGO}\exp\{-6t_{3}^{HCGO}/\theta _{HCGO}\}]\times [2/\theta _{HCGO}\exp\{-2t_{2}^{HCGO}/\theta _{HCGO}\}]\end{aligned}}$

Impact on phylogenetic estimation

The multispecies coalescent has profound implications for the theory and practice of molecular phylogenetics. Since individual gene trees can differ from the species tree one cannot estimate the tree for a single locus and assume that the gene tree correspond the species tree. In fact, one can be virtually certain that any individual gene tree will differ from the species tree for at least some relationships when any reasonable number of taxa are considered. However, gene tree-species tree discordance has an impact on the theory and practice of species tree estimation that goes beyond the simple observation that one cannot use a single gene tree to estimate the species tree because there is a part of parameter space where the most frequent gene tree is incongruent with the species tree. This part of parameter space is called the anomaly zone and any discordant gene trees that are more expected to arise more often than the gene tree. that matches the species tree are called anomalous gene trees.

The existence of the anomaly zone implies that one cannot simply estimate a large number of gene trees and assume the gene tree recovered the largest number of times is the species tree. Of course, estimating the species tree by a "democratic vote" of gene trees would only work for a limited number of taxa outside of the anomaly zone given the extremely large number of phylogenetic trees that are possible. However, the existence of the anomalous gene trees also means that simple methods for combining gene trees, like the majority rule extended ("greedy") consensus method or the matrix representation with parsimony (MRP) supertree approach, will not be consistent estimators of the species tree (i.e., they will be misleading). Simply generating the majority-rule consensus tree for the gene trees, where groups that are present in at least 50% of gene trees are retained, will not be misleading as long as a sufficient number of gene trees are used. However, this ability of the majority-rule consensus tree for a set of gene trees to avoid incorrect clades comes at the cost of having unresolved groups.

Simulations have shown that there are parts of species tree parameter space where maximum likelihood estimates of phylogeny are incorrect trees with increasing probability as the amount of data analyzed increases. This is important because the "concatenation approach," where multiple sequence alignments from different loci are concatenated to form a single large supermatrix alignment that is then used for maximum likelihood (or Bayesian MCMC) analysis, is both easy to implement and commonly used in empirical studies. This represents a case of model misspecification because the concatenation approach implicitly assumes that all gene trees have the same topology. Indeed, it has now been proven that analyses of data generated under the multispecies coalescent using maximum likelihood analysis of a concatenated data are not guaranteed to converge on the true species tree as the number of loci used for the analysis increases (i.e., maximum likelihood concatenation is is statistically inconsistent).

Software for inference using the multispecies coalescent

There are three basic methods for phylogenetic estimation in the multispecies coalescent framework: 1) full likelihood methods, which integrate over the uncertainty in gene trees (the approach originally proposed by Maddison 1997); 2) gene tree summary methods, which accept a set of gene trees as input and output an estimate of the species tree; and 3) site pattern methods, which produce an estimate of the species tree directly from aligned sites.

Software for phylogenetic estimation in the multispecies coalescent framework

Program	Description	Method	References
ASTRAL	ASTRAL (Accurate Species TRee ALgorithm) summarizes a set of gene trees using a quartet method generate an estimate of the species tree with coalescent branch lengths and support values (local posterior probabilities)	Summary	Mirarab et al. (2014); Zhang et al. (2018)
BPP	Software package for inferring phylogeny and divergence times among populations under the multispecies coalescent process; also includes method for species delimitation	Full likelihood	Yang et al. (2015); Flouri et al. (2018)
MP-EST	Accepts a set of gene trees as input and generates the maximum pseudolikelihood estimate of the species tree	Summary	Liu et al. (2010)
SVDquartets (in PAUP*)	PAUP* is a general phylogenetic estimation package that implements many methods. SVDquartets is a method that has shown to be statistically consistent for data generated given the multispecies coalescent	Site-pattern method	Chifman and Kubatko (2014)