She was the daughter of Charles Noel, 1st Earl of Gainsborough and Lady Frances Jocelyn, daughter of Robert Jocelyn, 3rd Earl of Roden. She spent her early years at the Noel family's ancestral home at Exton Hall, Rutland and later at Barham Court, Maidstone. She was educated by governess and travelled extensively on the continent. Her parents were of strong evangelical faith and placed great emphasis on community work in which Lady Victoria also participated.
In 1862 she married Sir Thomas Buxton, 3rd Baronet and resided with him at Warlies, Upshire near Waltham Abbey. The couple had thirteen children of whom ten survived. She assisted her husband with his political career and worked to support social services and church missions including the YMCA and YWCA.
In 1869 she began to suffer from osteoarthritis. Although she found travel difficult she remained reasonably active and retained her cheerful disposition. She held mothers’ meetings at Warlies and established a branch of the Church of England Mothers' Union. She served as diocesan president of the Mothers' Union in London where she was also president of the Time and Talents Association of young factory girls.
She was the founding president of the Mother’s Union of South Australia and actively supported the YWCA in Adelaide. She and her husband held a convention at which a constitution for a united Australia was first discussed. In 1896 she laid a foundation stone for a new junior branch of the YMCA in Adelaide, the Our Boys Institute. In 1898 Lady Victoria returned to England with her husband on leave. Her health deteriorated and she did not return to Australia. In 1902 the Buxtons built St Thomas's Church in Upshire.
She died at North Lodge, Cromer on 9 August 1916, aged 77.
- Joan B. Huffman, Buxton, Lady Victoria (1839–1916), Oxford Dictionary of National Biography, Oxford University Press, 2004