Phylogenomic Inference in the Presence of Gene Flow Using Coalescent Site Pattern Probabilities

• Activity Number: 159 - New Statistical Methods in Phylogenetics
• Type: Topic Contributed
• Date/Time: Monday, July 30, 2018 : 10:30 AM to 12:20 PM
• Sponsor: Biometrics Section
• Author(s): Laura Kubatko* and Colby Long
• Companies: Ohio State University and Mathematical Biosciences Institute, The Ohio State University
• Keywords: Species tree; phylogenetics; coalescent ; gene flow; gene tree

Abstract

Most current methods for inferring species phylogenies under the coalescent model assume that no gene flow occurs following speciation. We consider a three-taxon IM model that allows gene flow between sister taxa for a period of time following speciation, as well as variation in the effective population sizes across the tree. We derive the probabilities of the three gene tree topologies under the model, and show that for certain choices of the gene flow and effective population size parameters, anomalous gene trees (i.e., gene trees that are discordant with the species tree but which have higher probability than the gene tree concordant with the species tree) exist. We mathematically characterize the space over which there are anomalous trees, and show that the probability of the gene tree that matches the species tree can be made arbitrarily small. We then show that the site-patterns based method SVDQuartets is theoretically valid under this model. We use simulation to study its performance, and we compare it to other commonly-used methods for species tree inference.