Paine entered all three of the pistol events in the 1896 Games. He, along with his brother John Paine, was disqualified from the rapid fire pistol because their firearms were not of the appropriate caliber.
The Paine brothers used Colt revolvers in the military pistol event. These pistols were superior to the arms used by their opponents, and the brothers had little difficulty winning the top two spots. Sumner finished second with 380 points on 23 hits (of 30 shots) to John's 442 points on 25 hits. The next closest shooter (Nikolaos Morakis from Greece) scored only 205 points.
After winning the military pistol, John withdrew from the free pistol event. Sumner easily won this event as well, scoring exactly the same number of points (442) as John had to win the military pistol event. He did this on one fewer hit (24), though. In this case, the second place competitor scored 285 points.