Sun Jihai started his football career with Dalian Shide in 1995 and made his first appearance on 28 May 1995. Like many of the top young Chinese players of his generation, he wanted to go to the Chinese youth training camp in Brazil that was sponsored by Jianlibao. However, he was not selected as he was not deemed to possess enough potential. Sun showed great character by not letting this huge disappointment derail him and worked very hard to establish himself at his club. Sun enjoyed a successful spell with Dalian as the club won four league titles and one Chinese FA Cup title during his time there.
After three successful years at Dalian Shide, Sun along with Fan Zhiyi signed for Crystal Palace in 1998. They became the first ever Chinese players to play in the English leagues. Sun made his debut for Crystal Palace in a 3–0 defeat at Bury F.C. in the first leg of a Football League Cup match. He was recalled by Dalian who struggled heavily at the bottom of league in the 1999 season.
In 2002, Sun signed for Premier League side Manchester City from Dalian Shide for £2 million and became the first ever Chinese player to sign for the club. He made his debut for the club in a 4–2 win against Coventry City. His solid defense and dangerous attacking forays won over many City fans. He was voted the best player for the club in September 2002. In October 2002, Sun became the first Chinese footballer to score in the Premier League when he headed home the first goal in a 2–0 win over Birmingham City.
At the beginning of the 2004–05 season, Sun damaged his cruciate ligaments in a tackle with Chelsea striker Eiður Guðjohnsen and missed the rest of the season. After recuperating and following a strict physical regimen devised by his father, Sun re-installed his place as a regular in the club's starting lineup. In the 2006–07 season, Sun was set back yet again by another injury and he made his return on 10 February 2007, against Portsmouth which his side lost 2–1. With the arrival of Sven-Göran Eriksson as the new manager, Sun rarely played in the 2007–08 season. He was replaced by Vedran Ćorluka at right back and Michael Ball played as the defensive substitute instead of Sun.
On 2 July 2008, Sun signed a two-year deal with English Championship club Sheffield United. Sun only managed nineteen minutes in his first match for Sheffield United, getting sent off after picking up two yellows in a friendly. He went on to make his full debut in the opening fixture of the 2008–09 season against Birmingham City. He was sent off during a game against Coventry City after a lunging tackle from behind on opposing player Michael Mifsud. The Football Association charged him for his behavior after receiving a red card even with his past good disciplinary record and handed him an additional one match ban. Having played regularly until November he succumbed to an injury and subsequently failed to force his way back into the first team and made only a handful of appearances in the FA Cup in the second half of the season. Sun was then released from his contract in July 2009 with Sheffield United claiming he had "failed to settle" in Yorkshire.
In July 2009, Sun was loaned to Sheffield United's satellite team Chengdu Blades in the Chinese Super League, hoping to make an impact. He steadily established himself at Chengdu and helped them to achieve the club's best ever league position at seventh place. However, the club was relegated to China League One in the fallout of a match fixing scandal. He returned to Sheffield United after the loan was finished, but his contract ended and was left without a club to play for.
On 8 January 2010, Sun signed a two-year contract with fellow top tier side Shaanxi Chanba. He was promoted as the team captain immediately and he extended his contract for another two years in December 2011. At the beginning of the 2012 season, Sun followed the club when it decided to move to Guizhou and rename themselves Guizhou Renhe. He played nineteen league matches and five Chinese FA Cup matches in the 2012 season as the club achieved fourth place that league season and were runners-up in the cup which gained them entry into AFC Champions League for the first time in the club's history.
On 4 February 2015, Sun transferred to fellow top tier side Chongqing Lifan on a free transfer. He made his debut for the club on 8 March 2014 in a 3–0 loss against Beijing Guoan.
Sun made his international debut for Chinese national team on 6 December 1996 against Uzbekistan. During the 2002 FIFA World Cup in a group stage match against Costa Rica, Sun picked up an ankle injury after a tackle from behind by Mauricio Solís after just 17 minutes. Unable to play, Sun was replaced after 25 minutes and sat out the rest of group stage matches. On 7 June 2008, during 2010 FIFA World Cup qualification against Qatar, Sun was shown a straight red card for dissent while he was warming up as a substitute on the bench. China lost the match 1–0. He was given a ban of five matches by FIFA and did not receive any call-ups to national team after this incident.