Brown, who was born in Kennoway, Fife, started his professional playing career at East Fife, joining them in 1944 from his local side Kennoway. Brown made 62 league appearance for the Fifers, scoring 20 goals as well as numerous cup appearances. He left in December 1950.
Brown moved to Blackpool for a fee of £26,500 (then the largest fee received by a Scottish club), joining compatriots Jackie Mudie, Ewan Fenton, Hugh Kelly and goalkeeper George Farm, where he earned the nickname Bomber. At Blackpool he enjoyed success; however, he missed the Seasiders' 1951 FA Cup Final appearance after injuring his knee ten days earlier. He also missed the famous 1953 FA Cup Final victory after breaking his leg whilst scoring an 88th-minute winner in the quarter-final against Arsenal on 28 February 1953. Brown was inducted into the Hall of Fame at Bloomfield Road, when it was officially opened by former Blackpool player Jimmy Armfield in April 2006. Organised by the Blackpool Supporters Association, Blackpool fans around the world voted on their all-time heroes. Five players from each decade are inducted; Brown is in the 1950s.
While at Blackpool, Brown won a further eleven international caps, scoring against Wales at both the start of the 1952–53 and 1953–54 seasons. Brown was selected for the 22-man squad for the 1954 World Cup finals. The Scottish FA decided to take only 13 of the 22 to the finals. Brown was among those who travelled. Brown's final two games came in the finals against Austria and Uruguay.
Brown moved to Luton Town in February 1957, for a fee of £8,000, scoring on his league debut at Leeds. He scored five goals in the Hatters' successful 1958–59 FA Cup run, including scoring the only goal in their sixth-round victory over his former club, Blackpool. He at last played in a Wembley final, as Luton lost 2–1 to Nottingham Forest in the 1959 FA Cup Final. He scored 51 goals in 151 league appearances for Luton.
In August 1963, Brown joined then non-leagueWigan Athletic as player-manager, replacing Johnny Ball. He remained at Wigan until July 1966. In November 1966, he returned to Luton, this time as manager of a side struggling in the Fourth Division. He soon changed the fortunes of the Hatters, leading them to the Fourth Division title in 1968. They started the next season promisingly (and were eventually to finish in third place); however, Brown was sacked in December 1968 after it was discovered he had applied for the vacant managerial post at Leicester City.
He was not out of work for long, joining Torquay United as manager in January 1969. He kept the Gulls comfortably in Division Three during his first two-and-a-half seasons in charge (finishes of sixth, thirteenth and tenth), but after a poor start to the 1971–72 season, he was sacked in October 1971. Torquay would eventually be relegated later that season.
On 20 June 1972, Brown became manager of Bury, leading them to twelfth place in Division Four at the end of his first season in charge. The following season, 1973–74, Bury started brightly, and would eventually be promoted in fourth place; Brown, however, had left on 19 November 1973, to manage Nottingham Forest, and took them to seventh place by the end of the season. The following season, Forest started badly and Brown upset the fans by selling Duncan McKenzie to Brian Clough's Leeds United. He was sacked on 3 January 1975, shortly after a 2–0 home defeat to local rivals Notts County. Clough replaced Brown at Forest.
On 5 May 1976, Brown was appointed manager of another of his former playing clubs, Blackpool, who at the time were in the Second Division. The following season saw them battle for promotion and eventually finish in fifth place, missing out on a return to Division one by two points. During the next season the club were again challenging for promotion, but after an argument with chairman Billy Cartmell, on 6 February 1978 Brown was sacked. Cartmell had made remarks in a daily newspaper about Brown's job being on the line despite five-goal home wins in recent games, to which Brown responded by calling the chairman a "back-stabbing rat", which the board felt was unacceptable. After his departure, Blackpool won only one more game, dropped down the table, and were narrowly relegated at the end of the season.
Later that year he moved to Kuwait to manage Quadsia, but returned to manage Blackpool in March 1981 as a replacement for Alan Ball; however, he was unable to prevent their relegation to Division Four for the first time in their history. The following season saw a mid-table finish, and Brown left Blackpool in May 1982.
Brown died on 20 April 2011, at the age of 84. A wreath was laid behind the South Stand goal before the fixture between Blackpool and Newcastle United at Bloomfield Road three days after his death.