He was born in St. Paul, Minnesota, the third child of James J. and Johanna (Shiely) Duffy. When the death of the parents in 1879 orphaned the eight Duffy children, James Albert went to live at the Boys Orphan Asylum in Minneapolis. From 1887-1893 he was a student at St. Thomas College and subsequently attended St. Paul Seminary from 1894-1899. He was ordained on 27 May 1899 in St. Paul, Minnesota, by Rt Rev James Trober. Following his ordination Fr. Duffy served as a priest at Immaculate Conception Church in Minneapolis (1899-1902) and at St. Anne's Church in LeSueur, Minnesota (1902-1904). In 1904 he became rector at the cathedral parish of St. Mary's in Cheyenne, WY, under Bishop James J. Keane. Fr. Duffy was consecrated as the first bishop of the Diocese of Kearney (which became the Diocese of Grand Island in 1917) by Archbishop James J. Keane on April 16, 1913. Bishop Duffy oversaw construction of the Cathedral of St. Mary from 1926-1928, and he also established the Nebraska Register. He remained bishop until 1931, after which he became Titular Bishop of Silandus. He died on 12 February 1968 in Hot Springs, Arkansas, and was buried in Calvary Cemetery, Grand Island, Nebraska.
At the time of his death, Bishop Duffy was the senior Bishop in the United States in both age and years of consecration.