Born 12 February 1912 in Halifax, Yorkshire, the third child and second son of John and Margherita Whitley. Like his father, Oliver was a boarder in Wiseman's House, Clifton College where he was resident between 1925 and 1930. He went on to read History and Law at New College, Oxford and qualified as a Barrister. He married Elspeth Catherine Forrester-Paton in May 1939 and they had four sons and one daughter.
Oliver joined the BBC in 1935 and, at the outbreak of World War II moved to the Monitoring Service at Wood Norton monitoring Nazi broadcasts. He resigned in 1941 following a dispute over the proposal to move the unit to Caversham Park.