Phylogenetic bracketing is a method of inference used in biological sciences. It is to infer the likelihood of unknown traits in organisms based on their position in a phylogenetic tree. One of the main applications of phylogenetic bracketing is on extinct animals, known only from fossils. The method is often used for understanding traits that do not fossilize well, such as soft tissue anatomy, physiology and behaviour. By considering the closest and second closest well known (usually extant) organisms, traits can be asserted with a fair degree of certainty, though the method is extremely sensitive to problems from convergent evolution.
Extant Phylogenetic Bracketing requires that the species forming the brackets be extant. More general forms of phylogenetic bracketing do not require this and may use a mix of extant and extinct taxa to form the bracket. These more generalized forms of phylogenetic bracketing have the advantage in that they can be applied to a wider array of phylogenetic cases. However since these forms of bracketing are also more generalized and may rely on inferring traits in extinct animals, they also offer lower explanatory power compared to the EPB.
Extant phylogenetic bracketing (EPB)
This is a popular form of phylogenetic bracketing first introduced by Witmer in 1995. It works by comparing an extinct taxon to its nearest living relatives. For example, Tyrannosaurus, a theropod dinosaur, is bracketed by birds and crocodiles. A feature found in both birds and crocodiles would likely be present in Tyrannosaurus, such as the capability to lay an amniotic egg, whereas a feature both birds and crocodiles lack, such as hair, would probably not be present in Tyrannosaurus. Sometimes this approach is used for the reconstruction of ecological traits as well.
Levels of inference
The extant phylogenetic bracket approach allows researchers to infer traits in extinct animals with varying levels of confidence. This is referred to as the levels of inference. There are three levels of inference, with each higher level indicating less confidence for the inference.
Level 1 — The inference of a character that leaves a bony signature on the skeleton in both members of the extant sister groups. Example: Saying that Tyrannosaurus rex had an eyeball is a level 1 inference because both extant members of the group encompassing Tyrannosaurus rex have eyeballs, the homology of which is well established, and the eyes make noticeable impressions on the skull (orbital excavations).
Level 2 — The inference of a character that leaves a bony signature on the skeleton on only one side of the extant sister groups. For example saying that Tyrannosaurus rex had air sacs running through its skeleton is a level 2 inference as birds are the only extant sister group to Tyrannosaurus rex to show air sacs. However the excavation of the underlying bone (pneumatic fossae) by air sac diverticula in extant birds is remarkably similar to the cavities seen in the vertebrae of Tyrannosaurus rex, thus whereas the statement that Tyrannosaurus rex had air sacs is considered equivocal by EPB standards, the high degree of similarity between the pneumatic fossae in Tyrannosaurus rex and extant birds makes this a fairly strong inference.
Level 3 — The inference of a character that leaves a bony signature on the skeleton but is not present in either extant sister group to the taxon in question. For example saying that ceratopsian dinosaurs such as Triceratops horridus had horns in life would be a level 3 inference. Neither extant crocodylians, nor extant birds have horns today, but the osteological evidence for horns in ceratopsians is without question. Thus a level 3 inference receives no support from the EPB, but can still be used with confidence based on the merits of the fossil data itself.
Level 1′ — The inference of a character that is shared by both extant sister groups, but does not leave behind a bony signature. For example saying that Tyrannosaurus rex had a four-chambered heart would be a level 1′ inference as both extant sister groups (Crocodylia and Aves) have four-chambered hearts, but this trait does not leave behind any bony evidence.
Level 2′ — The inference of a character that is found in only one sister group to the taxon in question and that does not leave behind any bony evidence. For instance saying that Tyrannosaurus rex was warm-blooded would be a level 2′ inference as extant birds are warm-blooded but extant crocodylians are not. Further, since warm-bloodedness is a physiological trait rather than an anatomical one, it does not leave behind any bony signatures to indicate its presence.
Level 3′ — The inference of a character that is found in neither sister group to the taxon in question and that does not leave behind any bony signatures. For example saying that the large sauropod dinosaur Apatosaurus ajax gave birth to live young similar to mammals and many lizards  would be a level 3′ inference as neither crocodylians nor birds give birth to live young and these traits do not leave impressions on the skeleton.
In general the primes are always less confident than their underlying levels, however the confidence between levels is less clear cut. For instance it is unclear if a level 1′ would be less confident than a level 2. The same would go for a level 2′ vs. a level 3.
Example of bracketing with one extinct and one extant group
Assuming that the Kryptobaatar and Tachyglossidae spurs are homologous, they were a feature of their tribosphenidan last common ancestor, so we can tentatively conclude that they were present among the Early Cretaceous Eobaataridae—its descendants—as well.
Example of bracketing with only extinct groups
A fragmentary fossil with a known phylogeny can be compared to more complete fossil specimen to give an idea about general build and habit. The body of labyrinthodonts can usually be interfered to be broad and squat with a sideways compressed tail, although only the skull has been known for many taxa, based on the shape of more well-known labyrinthodont finds.
Example of failure using phylogenetic bracketing
Phylogenetic bracketing is based on the notion of anatomical conservationism. The general body shape of an animal can be fairly constant through large groups, but not always.
The large theropod dinosaur Spinosaurus was until 2014 only known from fragmentary remains, mainly of the skull and vertebrae. It was assumed that the remaining skeleton would look more or less like that of related animals like Baryonyx and Suchomimus, who sport a traditional theropod anatomy of long, strong hind legs and relatively small front legs. A 2014 find however, included a set of hind legs. The new reconstruction indicate earlier Spinosaurus reconstructions were wrong, and the animal was mainly aquatic and had relatively weak hind legs. It is possible it walked on all four when on land, the only theropod to do so.
- Witmer, L.M. 1995.The Extant Phylogenetic Bracket and the Importance of Reconstructing Soft Tissues in Fossils. in Thomason, J.J. (ed). Functional Morphology in Vertebrate Paleontology. New York. Cambridge University Press. pp: 19–33.
- Bryant, H.N. and Russell, A.P. 1992. The role of phylogenetic analysis in the inference of unpreserved attributes of extinct taxa. Philosophical Transactions of the Royal Society of London B 337:405-418.
- Witmer, L. M. 1998. "Application of the extant phylogenetic bracket (EPB) approach to the problem of anatomical novelty in the fossil record". Journal of Vertebrate Paleontology 18(3:Suppl.): 87A.
- Joyce, W. G. and Gauthier, J. A. 2003. Palaeoecology of Triassic stem turtles sheds new light on turtle origins. Proc. R. Soc. Lond. B (2004) 271: 1–5
- Bakker, R.T. 1986. The Dinosaur Heresies: New Theories Unlocking the Mystery of the Dinosaurs and their Extinction. New York. Kensington Publishing Corp.
- Kielen-Jaworowska, Zofia; Hurum, Jørn (2001). "Phylogeny and systematics of multituberculate mammals". Paleontology. 44, Part 3 (3): 389–429. doi:10.1111/1475-4983.00185. Retrieved October 25, 2013.
- Ibrahim, N.; Sereno, P. C.; Dal Sasso, C.; Maganuco, S.; Fabbri, M.; Martill, D. M.; Zouhri, S.; Myhrvold, N.; Iurino, D. A. (11 September 2014). "Semiaquatic adaptations in a giant predatory dinosaur". Science 345 (6204): 1613–1616. doi:10.1126/science.1258750. PMID 25213375.
- Switek, Brian (September 11, 2014). "The new Spinosaurus". Lealaps (National Geographic). Retrieved 21 November 2014.