Hårleman was born in Stockholm, son of the garden architect and head of the royal parks and gardens Johan Hårleman, who hade been ennobled in 1698, and began his architectural training under Göran Josua Adelcrantz. After receiving a state scholarship, he left Sweden for studies abroad in 1721, first going to Paris, where he spent four years as a student at the Royal French Academy of Architecture and the French Academy of Art. He later continued to Italy and was called back to Sweden while in Venice in 1727.
In 1728, upon the death of Nicodemus Tessin the Younger, Hårleman was appointed court intendant and subsequently in 1741, after Tessin's son Carl Gustaf Tessin had been made a member of the privy council, his successor as court superintendent. He was elected member of the Royal Swedish Academy of Sciences in 1744, was created a baron in 1747 and appointed Master of Ceremonies of the Royal Orders in 1748.
Hårleman completed the Royal Palace in Stockholm, begun by Nicodemus Tessin the younger after fire had destroyed the medieval castle, in 1697. He was particularly responsible for the interiors and employed a large number of qualified artisans for the work. The work on the interiors of the palace had a beneficial effect on the state of furniture-making and other crafts in Sweden and helped introduce the rococo style to the country.
Hårleman restored Uppsala Cathedral and parts of Uppsala Castle, both of which had been severely damaged in the Uppsala city fire of 1702, with the ruins of the castle having also been used as a quarry for the palace project in Stockholm. On behalf of Uppsala University, he built the Consistory House (konsistoriehuset) and the conservatory building for the botanical garden of Linnaeus.