Herrera led a group of Jewish refugees from Córdoba in 1474. Sefardic Gibraltar was granted to them by the Dukes of Medina Sidonia. The community grew to over 4,000, who paid taxes to Medina Sidonia and established a garrison of cavalry at his command. This lasted only 2 years though. In 1476, the Duke of Medina Sidonia realigned with Spain; the Sefardim were then forced back to Córdoba, many suffering persecution for heresy by the Inquisition.