Olliver began riding at the age of six, not uncommon for the times, before becoming a stable lad to his uncle, one Mr Page, and later progressing into racing over obstacles, falling in his first ride at Finchley.
Olliver was among the seventeen riders who participated in the first official running of the Grand National in 1839, finishing second on Seventy Four. He went on to ride in a record nineteen Nationals, a feat not equalled until 2014.
His first victory came in 1842 when he piloted Gaylad. He became the first dual winning rider the following year when he rode Vanguard. The latter was later presented to Olliver as a gift, and he held the horse in such high esteem that when Vanguard died, Olliver had his hide used to make a sofa, which today is the property of Aintree racecourse.
Olliver's final victory in the National came aboard the veteran Peter Simple in 1853. He told the owner before the race, "Sometimes he means it and I don't, sometimes I means it he don't but today we both mean it." but Olliver also had several near misses. In addition to the aforementioned Seventy Four, beaten three lengths, he finished as runner up on St Leger, 1847, beaten a length and The Curate, 1848, beaten half a length, thus missing a record six victories, which would have stood to this day, by four and a half lengths. In Olliver's other National rides, he was third once and failed to complete the course ten times.
Away from the course, Olliver was known for his swarthy dark looks, which led many to believe that his ancestry was either Spanish or Gypsy or even both. This led him to be known as 'Black' Tom. His interest in women, combined with his love of a party and with a sense of generosity, often saw him in grave financial difficulties. On one occasion, when he had been placed in the 'Pokey' (Debtors prison), he was visited by some cavalry officers. The story goes that when asked if there was anything they could provide for him, he replied, "Get me a damn good wall jumper." Olliver described himself as "Hopelessly insolvent." even going to the length of adding the second l to his original surname Oliver, claiming that "It is better to have an extra £ in hand." (L is the symbol for the British pound.)
Olliver's last National was in 1858, after which time he retired from the saddle and became the landlord of 'The Star' public house in Leamington. He soon grew to miss racing and set himself up as a trainer in Wroughton in Wiltshire. In the winter of 1873, he set about preparing his George Frederick for a crack at the 1874 Epsom Derby, but his health began to deteriorate and in January 1874 he died at the age of sixty-one. Olliver's head lad continued the preparations of the horse, who won the richest prize in English racing that summer.
- Sporting Magazine April 1840
- Gallant Sport-John Pinfold PP115
- A-Z of the Grand National-Cottrell & Armytage pp365