While IFK Mariehamn was formed in 1919, the sports club did not have a football section until the mid-1930s. Initially the team participated primarily in local tournaments on Åland, only sporadically playing other Finnish or Swedish teams. The team has participated in the Finnish football leagues since 1945.
Until the 1970s, IFK Mariehamn played primarily in the Finnish football divisions 3 and 4. The club reached a peak in 1975 and 1976, when the club first advanced to division 2 and then reached division 1 (Ykkönen) the following year. After two seasons in division 1, IFK Mariehamn was relegated to division 2, where it would remain up until the 2000s, except for a few seasons in division 3 during the early 1990s.
In 2003, IFK Mariehamn returned to division 1. After only one season in division 1, the club advanced, for the first time in its history to the premier division of Finnish football, the Veikkausliiga, for the 2005 season following qualyfing the October 2004 games against FC Jazz. In its first season in the Veikkausliiga, IFK Mariehamn finished 12th out of 14 teams. In 2006, the club finished in 5th place and the year after 6th after an impressive run of unbeaten matches during the autumn of 2007.
In addition to playing in the Veikkausliiga, IFK Mariehamn participates in local Åland tournaments, having won the Åland cup 40 times and the Åland football championships 42 times. In recent years, the club has been the most dominant football team on Åland, having in 2008 won its 11th and 15th straight titles in these two events, respectively.
In 2009, IFK Mariehamn started its first season as a fully professional football club. In 2015, the team won the Finnish Cup for the first time.
On 23 October 2016, IFK Mariehamn defeated FC Ilves 2-1 to secure the first ever Veikkausliiga Championship for the island club.