In the 1906 Summer Olympics at Athens, Pilgrim was an 11th-hour addition to the team. He travelled to Athens on his own, missing the wave that washed over the deck of the American team's ship in Gibraltar. The wave injured about half-dozen athletes aboard the deck, including one of the favorites in the 400 metres, Harry Hillman.
Pilgrim advanced to the final in the 400 metres, and was third before the final straight. On that stretch, he passed Wyndham Halswelle of Great Britain and Nigel Barker of Australia to win in time of 53.2 seconds. In the 800 metres, Pilgrim passed James Lightbody on the final lap and won by 2 feet. This medal does not appear in results or tables published by the International Olympic Committee, which retroactively downgraded the 1906 Summer Olympics and does not consider them to have been true "Games of the Olympiad."