A fireman, Goldsmith started his career in 1951 as a half forward flanker, having been recruited by South Melbourne from Spotswood.
In 1952, he was shifted to fullback; and, in 1955, he became the first ever (and so far, only) specialist fullback to win the Brownlow Medal.
To the astonishment of all, in 1956, South Melbourne decided to play its 1955 Brownlow medallist fullback at full-forward. He played at full-forward for the remainder of his career. In 1957, he was South Melbourne's leading goalkicker with 43 goals.