There had been a house at Whiteford since around the 13th century, owned by various families. The estate was sold to John Call in 1763. His son, also named John, became High Sheriff of Cornwall in 1771, and built Whiteford House in 1775 after returning from working in the East India Company. It was described as having "a handsome front" and as well as being lavishly designed, the house included a "looking-glass room" where a post boy was reportedly held prisoner by a parlour maid.
Little of the house survives. Some fragments are incorporated in the house of the agent of the home farm, i.e. a Tuscan Doric porch and some tripartite windows.[a] Call's descendants retained ownership of the house until 1870. The estate was briefly owned by the Montague family before being sold to the Prince of Wales in 1879 and the mansion was converted into an office for the Duchy of Cornwall, who sold off all its contents furnishings for a pittance.
The main house was demolished in 1913 while still in the ownership of the Duchy and some of the stone was used to build the nearby Duchy College. The stables and a garden folly survive, and the folly (called Whiteford Temple) survive, and Whiteford Temple is now owned by the Landmark Trust and let as a holiday cottage. There are Coade stone plaques on the exterior of the Temple.
- Pevsner (writing in 1950) notes the existence of an outbuilding divided into cottages, the stables with cupola and wings, a ghost of the layout of the grounds, a bridge and the garden temple "now a cattle byre".
- "Mad Dog McRea to play last year of doorstep proms". West Briton. 13 July 2010. Retrieved 12 August 2016.
- Joseph Polsue, ed. (1872). A Complete Parochial History of the County of Cornwall. W.Lake. p. 187.
- Pevsner, N. (1950). E. Radcliffe, ed. Cornwall (2nd ed.). Penguin (published 1970). p. 240.
- "Local News". The Cornishman (66). 16 October 1879. p. 3.
- "Whiteford Temple". Landmark Trust. Retrieved 12 August 2016.