He was educated at University College, Oxford where he matriculated in 1635. He then took up law at the Middle Temple and was called to the bar as a barrister in 1647.
His parents were John Packer, Clerk of the Privy Seal (12 Nov 1572 – 15 Feb 1649), and Philippi Mills.
He married Isabella Berkeley in 1653 in Groombridge, Kent; they had seven children:
- Robert Packer (died aged 16)
- John Philip Packer Esq. of Groombridge (1655 Groombridge - 16 December 1697)
- Catherine Packer (1655 Groombridge, Kent, England - 30 Nov 1722 Finedon, Northampton, England)
- Isabella Packer
- Elizabeth Packer
- Katherine Packer
- Temperance Packer
He married Sarah Isgar on 20 December 1666. Prior to their marriage, Philip and Sarah had three children in Ireland where Philip owned extensive properties:
- Philip Packer, who emigrated to New Jersey.
- James Packer (3 March 1657 – 12 Jul 1690) at the Battle of the Boyne, Ireland.
- William Packer (died 1690 in Battle of Boyne).
- Frommer's England 2010, Darwin Porter, Danforth Prince, Frommer's, 2009, ISBN 978-0-470-47070-1
- West Kent and the Weald, John Newman, Yale University Press, 1980, ISBN 978-0-300-09614-9
- Maidment, Barbara (2005). A History of Groombridge Place. Chelwood Gate: Danewood Press Ltd. p. 26. ISBN 0954120701.
- Maidment, Barbara (2001). A history of Groombridge Place. Brighton: Groombridge Past. p. 25. ISBN 0954120701.
- History of the Royal Society, Thomas Thomson
- "The Accompt of William Balle from 28 November 1660 to 11 September 1663", Notes and Records of the Royal Society of London, Vol. 14, No. 2 (Jun., 1960), pp. 174–183