|Representing the United States|
|1904 St. Louis||4 miles team race|
|1906 Athens||400 metres|
|1906 Athens||800 metres|
At the 1904 Summer Olympics in St. Louis, Missouri, Pilgrim competed in the 400 metre competition and 800 metre races, without success. He finished sixth in the four mile team run, aiding the New York Athletic Club (NYAC) to win the gold medal.
In the 1906 Summer Olympics at Athens, Pilgrim was an 11th-hour addition to the team. He travelled to Athens on his own, missing the wave that washed over the deck of the American team's ship in Gibraltar. The wave injured about half-dozen athletes aboard the deck, including one of the favorites in the 400 metres, Harry Hillman.
Pilgrim advanced to the final in the 400 metres, and was third before the final straight. On that stretch, he passed Wyndham Halswelle of Great Britain and Nigel Barker of Australia to win in time of 53.2 seconds. In the 800 metres, Pilgrim passed James Lightbody on the final lap and won by 2 feet. This medal does not appear in results or tables published by the International Olympic Committee, which retroactively downgraded the 1906 Summer Olympics and does not consider them to have been true "Games of the Olympiad."
Following his wins at the Athens Games, he never won a major competition again. He worked the rest of his career at NYAC, where one of his medals is on display.