In 1622, Vihterpalu estate was granted to Thomas von Ramm by the Swedish King Gustavus Adolphus. The present building probably dates from the 1830s and is in a late neoclassical style. The façade is dominated by a central pedimented section.
This land stayed in the wealthy von Ramm family until 1919, when the Republic of Estonia bought the manor in enforced procedures. In 1992, when the independence of Estonia was restored, the manor went back into the possession of Ramm family. Because of the deteriorated condition of the manor, the von Ramm family did not want it and gave it unconditionally to Estonia as a gift. The manor was bought by a group of Nordic businessmen in 1994, who wanted to renovate it, but died returning home by the passenger ferry MS Estonia, which sank on September 28, 1994. Eventually a Finnish businessman, Timo Lemberg, became the owner of the manor, and resumed its restoration.
Vihterpalu manor was restored in the beginning of 21st century. Now Vihterpalu Manor is a popular manor centre, offering an environment for weddings, conferences and social gatherings.