Robert was the son of Baron Sir Robert Ferrers of Wem and Elizabeth Boteler, 4th Baroness Boteler of Wem, who died in June 1411, and paternal grandson of Robert de Ferrers, 3rd Baron Ferrers of Chartley.
His father had been summoned to Parliament in 1375 as Robert Ferrers of Wem. Under modern peerage doctrine the manner in which he was named in this summons would be viewed as creating a novel peerage, the Barons Ferrers of Wem, to which his son Robert, who was never himself summoned, would be viewed to have succeeded as 2nd Baron on his father's death in 1380.
However, in Complete Peerage, Vicary Gibbs argues that contemporary practice was not so regimented as it would become, and that the elder Robert had clearly been summoned simply as possessor, jure uxoris, of the same barony previously held by his father-in-law William, Baron Boteler of Wem. His mother's 3rd husband, Sir Thomas Molinton, would in turn in his will style himself 'Lord of Wemme', jure uxoris, though he was never summoned. Were it the case that his father was summoned only jure uxoris, then Elizabeth's son Robert Ferrers, who was never himself summoned, would not have been a peer as he predeceased his mother.
Following this Robert's death in 1396 and of his mother in 1411, the Barony Boteler of Wem and any Barony Ferrers that might be held to have been created by the 1375 summons would have gone into abeyance between his two daughters.
- Elizabeth (1393–1434). She is buried at Black Friars Church, York. She married John de Greystoke, 4th Baron Greystoke (1389–1436) on 28 October 1407 in Greystoke Castle, Greystoke, Cumberland, and had issue.
- Mary or Margery (1394 – 25 January 1457/1458). She married her stepbrother, Sir Ralph Neville, son of Ralph Neville, 1st Earl of Westmoreland, before 1411 in Oversley, Warwickshire and had issue.
- Cokayne, George Edward (1912). Gibbs, Vicary (ed.). The Complete Peerage of England, Scotland, Ireland, Great Britain and the United Kingdom, extant, extinct or dormant. Vol. 2. London: The St. Catherine Press. pp. 232–233.
- Cokayne, George Edward (1916). Gibbs, Vicary; Doubleday, H. Arthur (eds.). The Complete Peerage of England, Scotland, Ireland, Great Britain and the United Kingdom, extant, extinct or dormant. Vol. 4. London: The St. Catherine Press. p. chart following 320.