From West Texas, Colbert had earned a reputation as a "gunfighter". He is said to have killed seven men in Texas, New Mexico, and Colorado. However, the only confirmed killing is that Charles Morris in Cimarron, New Mexico, a man he believed was involved romantically with his wife.
On January 7, 1874, Colbert and Allison entered the Clifton House, an Inn in Colfax County, New Mexico. They had finished a friendly quarter-mile horse race. Nevertheless, there was distrust between the men, since Allison was said to have mistreated Colbert's uncle, Zachary Colbert, a ferryman who had tried to overcharge Allison's family when they crossed the Brazos River. In the middle of their meal, Colbert suddenly tried to draw his revolver. Fortunately for Allison, the barrel stuck the table. Allison quickly drew and fired, strikinging Colbert in the head. Asked why he had accepted a dinner invitation from someone likely try to kill him, Allison replied: "Because I didn't want to send a man to hell on an empty stomach".
Colbert was buried in an unmarked grave behind the Clifton House.