Born in Colebrook, Connecticut, Phelps attended the public schools, the Episcopal Academy, Cheshire, Connecticut, Trinity College, Hartford, Connecticut, and the law department of Yale College. He was admitted to the bar in 1845 and commenced practice in Essex, Connecticut. He served as member of the State house of representatives in 1853, 1854, and 1856. He served in the State senate in 1858 and 1859. He served as judge of the superior court of Connecticut 1863-1873. He served as judge of the supreme court of errors of the State from 1873 until his resignation in 1875.
Phelps was elected as a Democrat to the Forty-fourth and to the three succeeding Congresses (March 4, 1875-March 3, 1883). He declined to be a candidate for renomination in 1882. He resumed the practice of law. He served as again judge of the superior court 1885-1892. He resumed the practice of his profession and also engaged in banking. He served as delegate to several State conventions. He died in Essex, Connecticut, January 15, 1900. He was interred in River View Cemetery.