There has been a manor house on the site of Keila-Joa manor (German: Schloss Fall) since the 17th century. The present manor house was built in 1831-1833 and designed by St. Petersburg architect Andrei Stackenschneider. The manor represents one of the earliest examples of neo-Gothic architecture in Estonia. It was built for the family of count Alexander von Benckendorff (whose graves can be found in the park adjacent to the manor) and the building saw many prominent guests during the Imperial years, among others the Russian royal family, famous soprano Henriette Sontag and composer Alexei Lvov.
From 1927 to 1940 it was used by the Estonian Ministry of Foreign Affairs. During the Soviet occupation it was used by the Red Army.