Sun Jihai started his football career with Dalian Shide in 1995 and made his first appearance on 28 May 1995. Like many of the top young Chinese players of the generation, he wanted to go to the Chinese youth training camp in that was sponsored by Jianlibao. However, he was not selected as he was not deemed to possess enough potential. Sun showed great character by not letting this huge disappointment derail him and worked very hard to establish himself at his club. Sun enjoyed a successful spell with Dalian as the club won four league titles and one Chinese FA Cup title during his time there.
After three successful years at Dalian, Sun along with Fan Zhiyi signed for Crystal Palace in 1998. They became the first footballers from China to play in the English leagues. Sun made his debut for Crystal Palace in a 3-0 defeat at Bury F.C. in the first leg of a Football League Cup match. He was recalled by Dalian Shide who struggled heavily in the bottom of league in 1999.
In 2002, Sun signed for Manchester City from Dalian for £2 million and became the first East Asian player to sign for Manchester City. He made his debut for the club in a 4–2 win over Coventry City. Sun enjoyed a great start to his first season in the Premier League. His solid defense and dangerous attacking forays won over many City fans. He was voted the best player at Manchester City for the month of September 2002. In October 2002, Sun became the first Chinese footballer to score in the Premier League when he headed home the first City goal in their 2–0 win over Birmingham City.
At the beginning of the 2004–05 season, Sun damaged his cruciate ligaments in a tackle with Chelsea striker Eiður Guðjohnsen and missed the rest of the season. After recuperating and following a strict physical regimen devised by his father, Sun re-installed his place as a regular starter in Manchester City's starting lineup. In the 2006-07 season, Sun was set back yet again by another injury and he made his return on 10 February 2007, against Portsmouth which his side lost 2-1. With the arrival of Sven-Göran Eriksson as the new manager, Sun rarely played in the 2007–08 season. He was replaced by Vedran Ćorluka at the right back position and Michael Ball played as the defensive substitute instead of him.
On 2 July 2008, Sun signed a two-year deal with English Championship club Sheffield United. Sun only managed nineteen minutes in his first match for Sheffield United, getting sent off after picking up two yellows in a friendly. He went on to make his full debut in the opening fixture of the 2008-09 season against Birmingham City. He was sent off during a game against Coventry City after a lunging tackle from behind on opposing player Michael Mifsud. The FA charged him for his behavior after receiving a red card even with his past good disciplinary record and handed him an additional one match ban. Having played regularly until November he succumbed to an injury and subsequently failed to force his way back into the first team and made only a handful of appearances in the FA Cup in the second half of the season. Sun was then released from his contract in July 2009 with Sheffield United claiming he had "failed to settle" in Yorkshire.
In July 2009, Sun was loaned to Sheffield United's affiliated club Chengdu Blades in the Chinese Super League, hoping to make an impact. He steadily established himself in Chengdu and helped them to achieve the club's best league position at seventh place. However, Chengdu Blades was relegated to China League One in the fallout of a match fixing scandal. He returned to Sheffield United after the loan was finished, but his contract ended and was left without a club to play for.
On 8 January 2010, Sun signed a two-year contract with another Chinese Super League club Shaanxi Chanba. He was promoted as the team captain immediately and he extended his contract for another two years in December 2011. Sun followed the club when it moved to Guiyang and changed their name to Guizhou Renhe in 2012. He played nineteen league matches and five Chinese FA Cup matches in the 2012 season as the club achieved fourth place that league season and were runners-up in the cup which gained them entry into AFC Champions League for the first time in the club's history.
Sun made his international debut for Chinese national team on 6 December 1996 against Uzbekistan. During the 2002 FIFA World Cup in a group stage match against Costa Rica, Sun picked up an ankle injury after a tackle from behind by Mauricio Solís after just 17 minutes. Unable to play, Sun was replaced after 25 minutes and sat out the rest of group stage matches. On 7 June 2008, during 2010 FIFA World Cup qualification against Qatar, Sun was shown a straight red card for dissent while he was warming up as a substitute on the bench. China lost the match 1-0. He was given a ban of five matches by FIFA and did not receive any call-ups to national team after this incident.