Kolovec was the site of two castles in the past. Both are mentioned and depicted by Johann Weikhard von Valvasor in his The Glory of the Duchy of Carniola as Gerlachstein. The older castle stood on a hill northeast of the settlement and was built in the 12th century. It was burned down in 1560 and the site was abandoned. The ruins of the castle are still visible. The second castle, built just outside the settlement to the southwest, was built in 1530. It burned down in 1679, but was rebuilt by the owners. In the Second World War it was used by the German army as its local headquarters until 1942, and was burned down by the Partisans in April 1943 to prevent the army from using it again. The remains are hardly visible today.