The present castle was built in 1180 by count Philip of Alsace  and was modeled after the crusaders castles that Philip of Alsace encountered while he participated in the second crusade. Before its construction, there stood a wooden castle on the same location, presumably built in the ninth century. The castle served as the seat of the Counts of Flanders until they abandoned it in the 14th century. The castle was then used as a courthouse, a prison and eventually decayed. Houses were built against the walls and even on the courtyard and the stones of the walls were used to erect other buildings. At one time it even served as a factory. At the end of the 19th century, the castle was scheduled to be demolished.
In 1885 the city of Ghent bought the castle and started a renovation project. The newly built houses were removed and the walls and keep were restored to their original condition.
The castle has been repaired enough to allow people to travel through it and climb on top. It is still partly surrounded by the moat. Inside is a museum with various torture devices (and a guillotine) that were historically used in Ghent.
The White Queen, a BBC drama series set in the 15th century, used the inside of the castle squares for some scenes and the outside view of the castle with the moat was shown in some short overall views (with some computer modifications to erase modern influences).